2024-03-29T10:39:51.145 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T10:39:51.145 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622657 --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --verbose --owner scheduled_amathuri@teuthology -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622657/orig.config.yaml 2024-03-29T10:39:51.181 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:39:51.271 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622657 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622657' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi112.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHuBDjejHQ6clU+zE35ohc++KSsObk4DDq5tqpmPWL13rsLQqLqicbLxqFFy7qS/SeaQoaT9UKuFHowzwksXC5w= smithi118.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJLpc8+kcufOIZ99A3ZlOO/UURp3bM65vBveLNBHcmJbrtsmZRV2it8LQ7naivtFd3Kr+YdOsKlHVhkdpc5FJHU= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T10:39:51.271 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T10:39:51.271 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T10:39:51.272 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T10:39:51.273 INFO:teuthology.task.internal:Checking packages... 2024-03-29T10:39:51.298 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T10:39:51.298 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T10:39:51.298 INFO:teuthology.packaging:ref: None 2024-03-29T10:39:51.298 INFO:teuthology.packaging:tag: None 2024-03-29T10:39:51.298 INFO:teuthology.packaging:branch: main 2024-03-29T10:39:51.298 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:39:51.298 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T10:39:51.546 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T10:39:51.547 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T10:39:51.554 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T10:39:51.554 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T10:39:51.560 INFO:teuthology.task.internal:Saving configuration 2024-03-29T10:39:51.573 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T10:39:51.579 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:39:51.603 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622657', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:34:18.768824', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHuBDjejHQ6clU+zE35ohc++KSsObk4DDq5tqpmPWL13rsLQqLqicbLxqFFy7qS/SeaQoaT9UKuFHowzwksXC5w='} 2024-03-29T10:39:51.627 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622657', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:34:18.766910', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJLpc8+kcufOIZ99A3ZlOO/UURp3bM65vBveLNBHcmJbrtsmZRV2it8LQ7naivtFd3Kr+YdOsKlHVhkdpc5FJHU='} 2024-03-29T10:39:51.628 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T10:39:51.635 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T10:39:51.635 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T10:39:51.635 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T10:39:51.707 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f68654770d0>, signals=[15]) 2024-03-29T10:39:51.708 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T10:39:51.714 INFO:teuthology.task.internal:Opening connections... 2024-03-29T10:39:51.714 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi112.front.sepia.ceph.com 2024-03-29T10:39:51.716 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:39:51.805 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi118.front.sepia.ceph.com 2024-03-29T10:39:51.807 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:39:51.935 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T10:39:51.942 DEBUG:teuthology.orchestra.run.smithi112:> uname -m 2024-03-29T10:39:51.959 INFO:teuthology.orchestra.run.smithi112.stdout:x86_64 2024-03-29T10:39:51.960 DEBUG:teuthology.orchestra.run.smithi112:> cat /etc/os-release 2024-03-29T10:39:52.016 INFO:teuthology.orchestra.run.smithi112.stdout:NAME="CentOS Stream" 2024-03-29T10:39:52.016 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION="9" 2024-03-29T10:39:52.016 INFO:teuthology.orchestra.run.smithi112.stdout:ID="centos" 2024-03-29T10:39:52.017 INFO:teuthology.orchestra.run.smithi112.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:39:52.017 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_ID="9" 2024-03-29T10:39:52.017 INFO:teuthology.orchestra.run.smithi112.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:39:52.017 INFO:teuthology.orchestra.run.smithi112.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:39:52.017 INFO:teuthology.orchestra.run.smithi112.stdout:ANSI_COLOR="0;31" 2024-03-29T10:39:52.017 INFO:teuthology.orchestra.run.smithi112.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:39:52.017 INFO:teuthology.orchestra.run.smithi112.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:39:52.017 INFO:teuthology.orchestra.run.smithi112.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:39:52.017 INFO:teuthology.orchestra.run.smithi112.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:39:52.017 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:39:52.017 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:39:52.018 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2024-03-29T10:39:52.039 DEBUG:teuthology.orchestra.run.smithi118:> uname -m 2024-03-29T10:39:52.056 INFO:teuthology.orchestra.run.smithi118.stdout:x86_64 2024-03-29T10:39:52.056 DEBUG:teuthology.orchestra.run.smithi118:> cat /etc/os-release 2024-03-29T10:39:52.113 INFO:teuthology.orchestra.run.smithi118.stdout:NAME="CentOS Stream" 2024-03-29T10:39:52.113 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION="9" 2024-03-29T10:39:52.113 INFO:teuthology.orchestra.run.smithi118.stdout:ID="centos" 2024-03-29T10:39:52.113 INFO:teuthology.orchestra.run.smithi118.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:39:52.113 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_ID="9" 2024-03-29T10:39:52.113 INFO:teuthology.orchestra.run.smithi118.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:39:52.113 INFO:teuthology.orchestra.run.smithi118.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:39:52.114 INFO:teuthology.orchestra.run.smithi118.stdout:ANSI_COLOR="0;31" 2024-03-29T10:39:52.114 INFO:teuthology.orchestra.run.smithi118.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:39:52.114 INFO:teuthology.orchestra.run.smithi118.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:39:52.114 INFO:teuthology.orchestra.run.smithi118.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:39:52.114 INFO:teuthology.orchestra.run.smithi118.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:39:52.114 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:39:52.114 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:39:52.114 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server 2024-03-29T10:39:52.139 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T10:39:52.148 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T10:39:52.154 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T10:39:52.154 DEBUG:teuthology.orchestra.run.smithi112:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:39:52.157 DEBUG:teuthology.orchestra.run.smithi118:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:39:52.172 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T10:39:52.178 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T10:39:52.178 DEBUG:teuthology.orchestra.run.smithi112:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:39:52.213 DEBUG:teuthology.orchestra.run.smithi118:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:39:52.309 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T10:39:52.364 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T10:39:52.378 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T10:39:52.378 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T10:39:52.378 DEBUG:teuthology.orchestra.run.smithi112:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:39:52.379 DEBUG:teuthology.orchestra.run.smithi118:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:39:52.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:39:52.395 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2024-03-29T10:39:52.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:39:52.396 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2024-03-29T10:39:52.450 INFO:teuthology.orchestra.run.smithi118.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:39:52.451 INFO:teuthology.task.kernel:Running kernel on smithi118: 5.14.0-361.el9.x86_64 2024-03-29T10:39:52.451 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum install -y kernel 2024-03-29T10:39:52.453 INFO:teuthology.orchestra.run.smithi112.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:39:52.454 INFO:teuthology.task.kernel:Running kernel on smithi112: 5.14.0-361.el9.x86_64 2024-03-29T10:39:52.454 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum install -y kernel 2024-03-29T10:39:54.516 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-29T10:39:55.463 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-29T10:39:56.201 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-03-29T10:39:57.030 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-9 - BaseOS 4.4 MB/s | 2.0 MB 00:00 2024-03-29T10:39:58.282 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.0 MB 00:01 2024-03-29T10:39:59.271 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.0 MB 00:01 2024-03-29T10:40:00.117 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-03-29T10:40:01.380 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-03-29T10:40:04.051 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-29T10:40:05.239 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.1 MB 00:00 2024-03-29T10:40:05.478 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-03-29T10:40:06.295 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 30 MB/s | 21 MB 00:00 2024-03-29T10:40:06.689 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 16 kB 00:00 2024-03-29T10:40:08.524 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-29T10:40:11.370 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-29T10:40:13.532 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-29T10:40:13.644 INFO:teuthology.orchestra.run.smithi112.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:40:13.644 INFO:teuthology.orchestra.run.smithi112.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:40:13.741 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:40:13.752 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:40:13.752 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-03-29T10:40:13.752 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:40:13.752 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-03-29T10:40:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:40:13.753 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-03-29T10:40:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:40:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:40:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:40:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:40:13.753 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-03-29T10:40:13.753 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:40:13.753 INFO:teuthology.orchestra.run.smithi112.stdout:Install 4 Packages 2024-03-29T10:40:13.753 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:40:13.754 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 76 M 2024-03-29T10:40:13.754 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 123 M 2024-03-29T10:40:13.754 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-03-29T10:40:13.905 INFO:teuthology.orchestra.run.smithi112.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T10:40:14.405 INFO:teuthology.orchestra.run.smithi112.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 23 MB/s | 15 MB 00:00 2024-03-29T10:40:15.105 INFO:teuthology.orchestra.run.smithi112.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 25 MB/s | 34 MB 00:01 2024-03-29T10:40:15.364 INFO:teuthology.orchestra.run.smithi112.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 19 MB/s | 28 MB 00:01 2024-03-29T10:40:15.367 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:40:15.367 INFO:teuthology.orchestra.run.smithi112.stdout:Total 47 MB/s | 76 MB 00:01 2024-03-29T10:40:15.368 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-03-29T10:40:15.718 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-03-29T10:40:15.718 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-03-29T10:40:15.797 INFO:teuthology.orchestra.run.smithi118.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:40:15.797 INFO:teuthology.orchestra.run.smithi118.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:40:15.927 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:40:15.938 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:40:15.939 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-29T10:40:15.939 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:40:15.939 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-29T10:40:15.939 INFO:teuthology.orchestra.run.smithi118.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:40:15.939 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-29T10:40:15.939 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:40:15.939 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:40:15.939 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:40:15.939 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:40:15.939 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-29T10:40:15.939 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:40:15.939 INFO:teuthology.orchestra.run.smithi118.stdout:Install 4 Packages 2024-03-29T10:40:15.940 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:40:15.940 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 76 M 2024-03-29T10:40:15.940 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 123 M 2024-03-29T10:40:15.940 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-29T10:40:16.109 INFO:teuthology.orchestra.run.smithi118.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-29T10:40:16.578 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-03-29T10:40:16.578 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-03-29T10:40:16.592 INFO:teuthology.orchestra.run.smithi118.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 23 MB/s | 15 MB 00:00 2024-03-29T10:40:17.134 INFO:teuthology.orchestra.run.smithi118.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 28 MB/s | 34 MB 00:01 2024-03-29T10:40:17.834 INFO:teuthology.orchestra.run.smithi118.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 16 MB/s | 28 MB 00:01 2024-03-29T10:40:17.837 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:40:17.837 INFO:teuthology.orchestra.run.smithi118.stdout:Total 40 MB/s | 76 MB 00:01 2024-03-29T10:40:17.838 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-29T10:40:18.199 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-29T10:40:18.199 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-29T10:40:18.887 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-03-29T10:40:19.031 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-29T10:40:19.031 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-29T10:40:19.477 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:40:19.556 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:40:21.338 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-29T10:40:21.964 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:40:22.092 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:40:22.323 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:40:22.458 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:40:24.833 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:40:24.969 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:40:27.142 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:40:27.265 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:40:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:40:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:40:31.969 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:40:34.457 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:41:01.054 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:41:01.054 INFO:teuthology.orchestra.run.smithi112.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:41:01.054 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:41:01.098 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:41:01.501 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:41:01.501 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:41:01.501 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:41:01.502 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:41:01.911 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:41:01.911 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:41:01.911 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-03-29T10:41:01.911 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:41:01.911 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:41:01.911 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:41:01.911 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:41:01.911 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:41:01.912 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:41:02.223 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:41:02.328 INFO:teuthology.orchestra.run.smithi112.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:41:02.328 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:41:02.328 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:41:02.328 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:41:02.328 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:41:02.328 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T10:41:02.329 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi112.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:41:02.329 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:41:02.408 INFO:teuthology.orchestra.run.smithi112.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:41:02.409 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:41:02.409 DEBUG:teuthology.orchestra.run.smithi112:> sudo rpm -qi grub2-tools 2024-03-29T10:41:02.443 INFO:teuthology.orchestra.run.smithi112.stdout:Name : grub2-tools 2024-03-29T10:41:02.443 INFO:teuthology.orchestra.run.smithi112.stdout:Epoch : 1 2024-03-29T10:41:02.443 INFO:teuthology.orchestra.run.smithi112.stdout:Version : 2.06 2024-03-29T10:41:02.443 INFO:teuthology.orchestra.run.smithi112.stdout:Release : 68.el9 2024-03-29T10:41:02.443 INFO:teuthology.orchestra.run.smithi112.stdout:Architecture: x86_64 2024-03-29T10:41:02.443 INFO:teuthology.orchestra.run.smithi112.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:41:02.443 INFO:teuthology.orchestra.run.smithi112.stdout:Group : Unspecified 2024-03-29T10:41:02.443 INFO:teuthology.orchestra.run.smithi112.stdout:Size : 8264254 2024-03-29T10:41:02.444 INFO:teuthology.orchestra.run.smithi112.stdout:License : GPLv3+ 2024-03-29T10:41:02.444 INFO:teuthology.orchestra.run.smithi112.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:41:02.444 INFO:teuthology.orchestra.run.smithi112.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:41:02.444 INFO:teuthology.orchestra.run.smithi112.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:41:02.444 INFO:teuthology.orchestra.run.smithi112.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:41:02.444 INFO:teuthology.orchestra.run.smithi112.stdout:Packager : builder@centos.org 2024-03-29T10:41:02.444 INFO:teuthology.orchestra.run.smithi112.stdout:Vendor : CentOS 2024-03-29T10:41:02.444 INFO:teuthology.orchestra.run.smithi112.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:41:02.444 INFO:teuthology.orchestra.run.smithi112.stdout:Summary : Support tools for GRUB. 2024-03-29T10:41:02.444 INFO:teuthology.orchestra.run.smithi112.stdout:Description : 2024-03-29T10:41:02.444 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:41:02.444 INFO:teuthology.orchestra.run.smithi112.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:41:02.444 INFO:teuthology.orchestra.run.smithi112.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:41:02.444 INFO:teuthology.orchestra.run.smithi112.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:41:02.445 INFO:teuthology.orchestra.run.smithi112.stdout:hardware devices. 2024-03-29T10:41:02.445 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:41:02.445 INFO:teuthology.orchestra.run.smithi112.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:41:02.446 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:41:02.446 INFO:teuthology.task.kernel:Updating grub on smithi112 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:41:02.446 DEBUG:teuthology.orchestra.run.smithi112:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:41:03.025 INFO:teuthology.orchestra.run.smithi112.stderr:Generating grub configuration file ... 2024-03-29T10:41:04.356 INFO:teuthology.orchestra.run.smithi112.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:41:04.388 INFO:teuthology.orchestra.run.smithi112.stderr:done 2024-03-29T10:41:04.389 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2024-03-29T10:41:04.409 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.i3iWbrLWeX 2024-03-29T10:41:04.409 DEBUG:teuthology.orchestra.run.smithi112:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.i3iWbrLWeX 2024-03-29T10:41:04.486 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.i3iWbrLWeX 2024-03-29T10:41:04.629 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.i3iWbrLWeX is 6KB 2024-03-29T10:41:04.655 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.i3iWbrLWeX 2024-03-29T10:41:04.669 DEBUG:teuthology.orchestra.run.smithi112:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:41:04.747 INFO:teuthology.orchestra.run.smithi112.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:41:04.747 INFO:teuthology.orchestra.run.smithi112.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:41:04.747 INFO:teuthology.orchestra.run.smithi112.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:41:04.747 INFO:teuthology.orchestra.run.smithi112.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:41:04.748 DEBUG:teuthology.orchestra.run.smithi112:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:41:04.917 DEBUG:teuthology.orchestra.run.smithi112:> sudo shutdown -r now 2024-03-29T10:41:04.928 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:41:04.929 INFO:teuthology.orchestra.run.smithi118.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:41:04.929 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:41:04.972 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:41:05.337 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:41:05.337 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:41:05.337 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:41:05.337 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:41:05.752 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:41:05.753 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:41:05.753 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-29T10:41:05.753 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:41:05.753 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:41:05.753 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:41:05.753 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:41:05.753 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:41:05.753 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:41:06.103 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:41:06.200 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:41:06.200 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:41:06.200 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:41:06.200 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:41:06.200 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:41:06.201 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T10:41:06.201 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi118.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:41:06.201 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:41:06.280 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:41:06.280 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:41:06.281 DEBUG:teuthology.orchestra.run.smithi118:> sudo rpm -qi grub2-tools 2024-03-29T10:41:06.320 INFO:teuthology.orchestra.run.smithi118.stdout:Name : grub2-tools 2024-03-29T10:41:06.320 INFO:teuthology.orchestra.run.smithi118.stdout:Epoch : 1 2024-03-29T10:41:06.320 INFO:teuthology.orchestra.run.smithi118.stdout:Version : 2.06 2024-03-29T10:41:06.320 INFO:teuthology.orchestra.run.smithi118.stdout:Release : 68.el9 2024-03-29T10:41:06.320 INFO:teuthology.orchestra.run.smithi118.stdout:Architecture: x86_64 2024-03-29T10:41:06.320 INFO:teuthology.orchestra.run.smithi118.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:41:06.320 INFO:teuthology.orchestra.run.smithi118.stdout:Group : Unspecified 2024-03-29T10:41:06.320 INFO:teuthology.orchestra.run.smithi118.stdout:Size : 8264254 2024-03-29T10:41:06.320 INFO:teuthology.orchestra.run.smithi118.stdout:License : GPLv3+ 2024-03-29T10:41:06.320 INFO:teuthology.orchestra.run.smithi118.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:41:06.320 INFO:teuthology.orchestra.run.smithi118.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:41:06.320 INFO:teuthology.orchestra.run.smithi118.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:41:06.320 INFO:teuthology.orchestra.run.smithi118.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:41:06.321 INFO:teuthology.orchestra.run.smithi118.stdout:Packager : builder@centos.org 2024-03-29T10:41:06.321 INFO:teuthology.orchestra.run.smithi118.stdout:Vendor : CentOS 2024-03-29T10:41:06.321 INFO:teuthology.orchestra.run.smithi118.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:41:06.321 INFO:teuthology.orchestra.run.smithi118.stdout:Summary : Support tools for GRUB. 2024-03-29T10:41:06.321 INFO:teuthology.orchestra.run.smithi118.stdout:Description : 2024-03-29T10:41:06.321 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:41:06.321 INFO:teuthology.orchestra.run.smithi118.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:41:06.321 INFO:teuthology.orchestra.run.smithi118.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:41:06.321 INFO:teuthology.orchestra.run.smithi118.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:41:06.321 INFO:teuthology.orchestra.run.smithi118.stdout:hardware devices. 2024-03-29T10:41:06.321 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:41:06.321 INFO:teuthology.orchestra.run.smithi118.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:41:06.322 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:41:06.322 INFO:teuthology.task.kernel:Updating grub on smithi118 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:41:06.322 DEBUG:teuthology.orchestra.run.smithi118:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:41:06.918 INFO:teuthology.orchestra.run.smithi118.stderr:Generating grub configuration file ... 2024-03-29T10:41:08.287 INFO:teuthology.orchestra.run.smithi118.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:41:08.320 INFO:teuthology.orchestra.run.smithi118.stderr:done 2024-03-29T10:41:08.322 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2024-03-29T10:41:08.338 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.9cFmzxREGP 2024-03-29T10:41:08.338 DEBUG:teuthology.orchestra.run.smithi118:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.9cFmzxREGP 2024-03-29T10:41:08.418 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.9cFmzxREGP 2024-03-29T10:41:08.559 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.9cFmzxREGP is 6KB 2024-03-29T10:41:08.606 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.9cFmzxREGP 2024-03-29T10:41:08.621 DEBUG:teuthology.orchestra.run.smithi118:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:41:08.685 INFO:teuthology.orchestra.run.smithi118.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:41:08.685 INFO:teuthology.orchestra.run.smithi118.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:41:08.685 INFO:teuthology.orchestra.run.smithi118.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:41:08.685 INFO:teuthology.orchestra.run.smithi118.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:41:08.686 DEBUG:teuthology.orchestra.run.smithi118:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:41:08.841 DEBUG:teuthology.orchestra.run.smithi118:> sudo shutdown -r now 2024-03-29T10:41:34.947 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T10:41:34.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-03-29T10:41:34.948 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:41:38.846 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T10:41:38.846 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-29T10:41:38.847 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:41:53.438 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-03-29T10:41:57.278 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-03-29T10:42:02.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-03-29T10:42:02.443 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:42:05.502 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-03-29T10:42:06.280 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-29T10:42:06.281 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:42:09.342 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-03-29T10:42:17.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-03-29T10:42:17.512 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:42:21.346 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-29T10:42:21.347 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:42:49.300 DEBUG:teuthology.orchestra.run.smithi112:> true 2024-03-29T10:42:49.708 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-03-29T10:42:49.709 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:42:49.709 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2024-03-29T10:42:49.723 INFO:teuthology.orchestra.run.smithi112.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:42:49.725 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:42:49.725 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:42:49.726 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:42:50.726 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T10:42:50.726 DEBUG:teuthology.orchestra.run.smithi112:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:42:50.815 INFO:teuthology.orchestra.run.smithi112.stdout:ttyS1 2024-03-29T10:42:50.853 DEBUG:teuthology.parallel:result is None 2024-03-29T10:43:21.349 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T10:43:36.349 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-29T10:43:36.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:43:36.757 DEBUG:teuthology.orchestra.run.smithi118:> true 2024-03-29T10:43:37.146 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-29T10:43:37.146 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:43:37.146 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2024-03-29T10:43:37.161 INFO:teuthology.orchestra.run.smithi118.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:43:37.161 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:43:37.162 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:43:37.162 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:43:38.162 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T10:43:38.162 DEBUG:teuthology.orchestra.run.smithi118:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:43:38.246 INFO:teuthology.orchestra.run.smithi118.stdout:ttyS1 2024-03-29T10:43:38.276 DEBUG:teuthology.parallel:result is None 2024-03-29T10:43:38.277 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T10:43:38.284 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T10:43:38.284 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:43:38.287 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:43:38.304 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T10:43:38.310 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T10:43:38.320 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T10:43:38.321 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:43:38.345 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:43:38.424 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T10:43:38.430 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T10:43:38.431 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:43:38.433 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:43:38.467 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:43:38.472 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:43:38.481 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:43:38.486 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:43:38.488 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T10:43:38.495 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T10:43:38.496 DEBUG:teuthology.orchestra.run.smithi112:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:43:38.530 DEBUG:teuthology.orchestra.run.smithi118:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:43:38.559 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T10:43:38.569 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T10:43:38.569 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:43:38.600 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:43:38.616 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:43:38.695 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:43:38.744 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:43:38.825 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:43:38.870 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-03-29T10:43:38.870 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:43:38.937 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:43:38.977 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:43:39.032 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:43:39.110 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:43:39.160 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-29T10:43:39.160 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:43:39.230 DEBUG:teuthology.orchestra.run.smithi112:> sudo service rsyslog restart 2024-03-29T10:43:39.232 DEBUG:teuthology.orchestra.run.smithi118:> sudo service rsyslog restart 2024-03-29T10:43:39.277 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:43:39.329 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:43:39.755 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T10:43:39.763 INFO:teuthology.task.internal:Starting timer... 2024-03-29T10:43:39.763 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T10:43:39.773 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T10:43:39.781 DEBUG:teuthology.orchestra.run.smithi112:> sudo service auditd rotate 2024-03-29T10:43:39.846 INFO:teuthology.orchestra.run.smithi112.stdout:Rotating logs: 2024-03-29T10:43:39.848 DEBUG:teuthology.orchestra.run.smithi118:> sudo service auditd rotate 2024-03-29T10:43:39.908 INFO:teuthology.orchestra.run.smithi118.stdout:Rotating logs: 2024-03-29T10:43:39.910 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T10:43:39.910 DEBUG:teuthology.orchestra.run.smithi112:> /usr/sbin/getenforce 2024-03-29T10:43:39.933 INFO:teuthology.orchestra.run.smithi112.stdout:Permissive 2024-03-29T10:43:39.933 DEBUG:teuthology.orchestra.run.smithi118:> /usr/sbin/getenforce 2024-03-29T10:43:39.963 INFO:teuthology.orchestra.run.smithi118.stdout:Permissive 2024-03-29T10:43:39.964 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi112.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi118.front.sepia.ceph.com': 'permissive'} 2024-03-29T10:43:39.964 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:43:40.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:43:40.002 DEBUG:teuthology.orchestra.run.smithi118:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:43:40.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:43:40.031 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T10:43:40.031 DEBUG:teuthology.orchestra.run.smithi112:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:43:40.077 DEBUG:teuthology.orchestra.run.smithi118:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:43:40.132 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T10:43:40.142 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-29T10:43:40.148 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-29T10:43:40.297 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T10:43:40.313 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T10:43:40.314 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi112.front.sepia.ceph.com,smithi118.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T10:47:57.652 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi112.front.sepia.ceph.com'), Remote(name='ubuntu@smithi118.front.sepia.ceph.com')] 2024-03-29T10:47:57.653 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-03-29T10:47:57.654 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:47:57.742 DEBUG:teuthology.orchestra.run.smithi112:> true 2024-03-29T10:47:57.822 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-03-29T10:47:57.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-29T10:47:57.823 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:47:57.908 DEBUG:teuthology.orchestra.run.smithi118:> true 2024-03-29T10:47:57.989 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-29T10:47:57.989 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T10:47:57.999 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T10:47:57.999 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:47:57.999 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:47:58.001 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:47:58.002 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:47:58.032 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:47:58.049 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:47:58.060 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:47:58.076 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:47:58.079 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: ntpd: command not found 2024-03-29T10:47:58.093 INFO:teuthology.orchestra.run.smithi112.stdout:506 Cannot talk to daemon 2024-03-29T10:47:58.106 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: ntpd: command not found 2024-03-29T10:47:58.109 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:47:58.119 INFO:teuthology.orchestra.run.smithi118.stdout:506 Cannot talk to daemon 2024-03-29T10:47:58.125 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:47:58.136 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:47:58.152 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:47:58.176 INFO:teuthology.orchestra.run.smithi112.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:47:58.179 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:47:58.179 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2024-03-29T10:47:58.179 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:47:58.179 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:47:58.179 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:47:58.179 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:47:58.209 INFO:teuthology.orchestra.run.smithi118.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:47:58.212 INFO:teuthology.orchestra.run.smithi118.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:47:58.212 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================== 2024-03-29T10:47:58.213 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:47:58.213 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:47:58.213 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:47:58.213 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:47:58.214 INFO:teuthology.run_tasks:Running task install... 2024-03-29T10:47:58.224 DEBUG:teuthology.task.install:project ceph 2024-03-29T10:47:58.224 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T10:47:58.224 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T10:47:58.224 INFO:teuthology.task.install:Using flavor: default 2024-03-29T10:47:58.230 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T10:47:58.230 INFO:teuthology.task.install:extra packages: [] 2024-03-29T10:47:58.231 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:47:58.231 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:47:58.233 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:47:58.233 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:47:58.384 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:47:58.385 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:47:58.447 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:47:58.447 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:47:58.806 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:47:58.806 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-29T10:47:58.806 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:47:58.835 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:47:58.836 DEBUG:teuthology.orchestra.run.smithi118:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:47:58.847 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:47:58.847 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-03-29T10:47:58.847 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:47:58.876 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:47:58.876 DEBUG:teuthology.orchestra.run.smithi112:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:47:58.905 DEBUG:teuthology.orchestra.run.smithi118:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:47:58.945 DEBUG:teuthology.orchestra.run.smithi112:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:47:58.987 DEBUG:teuthology.orchestra.run.smithi118:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:47:59.027 DEBUG:teuthology.orchestra.run.smithi112:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:47:59.054 INFO:teuthology.orchestra.run.smithi118.stdout:check_obsoletes = 1 2024-03-29T10:47:59.055 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum clean all 2024-03-29T10:47:59.094 INFO:teuthology.orchestra.run.smithi112.stdout:check_obsoletes = 1 2024-03-29T10:47:59.095 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum clean all 2024-03-29T10:47:59.302 INFO:teuthology.orchestra.run.smithi118.stdout:74 files removed 2024-03-29T10:47:59.332 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-radosgw 2024-03-29T10:47:59.343 INFO:teuthology.orchestra.run.smithi112.stdout:74 files removed 2024-03-29T10:47:59.371 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-radosgw 2024-03-29T10:48:00.563 INFO:teuthology.orchestra.run.smithi118.stdout:ceph packages for x86_64 88 kB/s | 82 kB 00:00 2024-03-29T10:48:00.625 INFO:teuthology.orchestra.run.smithi112.stdout:ceph packages for x86_64 86 kB/s | 82 kB 00:00 2024-03-29T10:48:01.298 INFO:teuthology.orchestra.run.smithi118.stdout:ceph noarch packages 15 kB/s | 10 kB 00:00 2024-03-29T10:48:01.384 INFO:teuthology.orchestra.run.smithi112.stdout:ceph noarch packages 14 kB/s | 10 kB 00:00 2024-03-29T10:48:01.973 INFO:teuthology.orchestra.run.smithi118.stdout:ceph source packages 3.0 kB/s | 1.9 kB 00:00 2024-03-29T10:48:02.102 INFO:teuthology.orchestra.run.smithi112.stdout:ceph source packages 2.8 kB/s | 1.9 kB 00:00 2024-03-29T10:48:02.384 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-29T10:48:02.570 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-29T10:48:03.842 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2024-03-29T10:48:04.097 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS-9 - BaseOS 4.9 MB/s | 2.0 MB 00:00 2024-03-29T10:48:05.130 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-29T10:48:05.441 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - BaseOS 6.3 MB/s | 8.0 MB 00:01 2024-03-29T10:48:07.470 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-29T10:48:08.206 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - AppStream 9.4 MB/s | 19 MB 00:02 2024-03-29T10:48:12.193 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 6.1 MB 00:01 2024-03-29T10:48:12.417 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - CRB 8.4 MB/s | 6.1 MB 00:00 2024-03-29T10:48:13.586 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 16 kB 00:00 2024-03-29T10:48:13.974 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 16 kB 00:00 2024-03-29T10:48:14.630 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-29T10:48:15.126 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-29T10:48:19.713 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-29T10:48:20.142 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-29T10:48:22.407 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:48:22.426 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:48:22.426 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-29T10:48:22.426 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:48:22.426 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-29T10:48:22.427 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:48:22.427 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrading: 2024-03-29T10:48:22.427 INFO:teuthology.orchestra.run.smithi118.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:48:22.427 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:48:22.427 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-29T10:48:22.427 INFO:teuthology.orchestra.run.smithi118.stdout: boost-program-options 2024-03-29T10:48:22.427 INFO:teuthology.orchestra.run.smithi118.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:48:22.427 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:48:22.427 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:48:22.427 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:48:22.427 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:48:22.427 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:48:22.427 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:48:22.428 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:48:22.428 INFO:teuthology.orchestra.run.smithi118.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:48:22.428 INFO:teuthology.orchestra.run.smithi118.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:48:22.428 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:48:22.428 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:48:22.428 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:48:22.428 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:48:22.428 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:48:22.428 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:48:22.428 INFO:teuthology.orchestra.run.smithi118.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:48:22.428 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse 2024-03-29T10:48:22.428 INFO:teuthology.orchestra.run.smithi118.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:48:22.428 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:48:22.428 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:48:22.429 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:48:22.429 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:48:22.429 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:48:22.429 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:48:22.429 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:48:22.429 INFO:teuthology.orchestra.run.smithi118.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:48:22.429 INFO:teuthology.orchestra.run.smithi118.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:48:22.429 INFO:teuthology.orchestra.run.smithi118.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:48:22.429 INFO:teuthology.orchestra.run.smithi118.stdout:Installing weak dependencies: 2024-03-29T10:48:22.429 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:48:22.429 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:48:22.429 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-29T10:48:22.429 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:48:22.429 INFO:teuthology.orchestra.run.smithi118.stdout:Install 30 Packages 2024-03-29T10:48:22.430 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrade 2 Packages 2024-03-29T10:48:22.430 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:48:22.430 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 58 M 2024-03-29T10:48:22.430 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-29T10:48:22.732 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:48:22.752 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:48:22.753 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-03-29T10:48:22.753 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:48:22.753 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-03-29T10:48:22.753 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:48:22.753 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrading: 2024-03-29T10:48:22.753 INFO:teuthology.orchestra.run.smithi112.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:48:22.753 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:48:22.753 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-03-29T10:48:22.753 INFO:teuthology.orchestra.run.smithi112.stdout: boost-program-options 2024-03-29T10:48:22.753 INFO:teuthology.orchestra.run.smithi112.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:48:22.753 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:48:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:48:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:48:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:48:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:48:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:48:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:48:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:48:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:48:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:48:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:48:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:48:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:48:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:48:22.754 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:48:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:48:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse 2024-03-29T10:48:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:48:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:48:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:48:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:48:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:48:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:48:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:48:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:48:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:48:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:48:22.755 INFO:teuthology.orchestra.run.smithi112.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:48:22.755 INFO:teuthology.orchestra.run.smithi112.stdout:Installing weak dependencies: 2024-03-29T10:48:22.756 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:48:22.756 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:48:22.756 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-03-29T10:48:22.756 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:48:22.756 INFO:teuthology.orchestra.run.smithi112.stdout:Install 30 Packages 2024-03-29T10:48:22.756 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrade 2 Packages 2024-03-29T10:48:22.756 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:48:22.756 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 58 M 2024-03-29T10:48:22.756 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-03-29T10:48:23.889 INFO:teuthology.orchestra.run.smithi118.stdout:(1/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 16 MB/s | 20 MB 00:01 2024-03-29T10:48:23.939 INFO:teuthology.orchestra.run.smithi118.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 506 kB/s | 25 kB 00:00 2024-03-29T10:48:24.039 INFO:teuthology.orchestra.run.smithi118.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 7.3 MB/s | 742 kB 00:00 2024-03-29T10:48:24.115 INFO:teuthology.orchestra.run.smithi118.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 6.5 MB/s | 498 kB 00:00 2024-03-29T10:48:24.507 INFO:teuthology.orchestra.run.smithi118.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 13 MB/s | 5.1 MB 00:00 2024-03-29T10:48:24.557 INFO:teuthology.orchestra.run.smithi118.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 906 kB/s | 45 kB 00:00 2024-03-29T10:48:24.616 INFO:teuthology.orchestra.run.smithi118.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 2.2 MB/s | 131 kB 00:00 2024-03-29T10:48:24.662 INFO:teuthology.orchestra.run.smithi112.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 3.0 MB/s | 5.3 MB 00:01 2024-03-29T10:48:24.674 INFO:teuthology.orchestra.run.smithi118.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.7 MB/s | 162 kB 00:00 2024-03-29T10:48:24.710 INFO:teuthology.orchestra.run.smithi112.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 505 kB/s | 25 kB 00:00 2024-03-29T10:48:24.741 INFO:teuthology.orchestra.run.smithi118.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 4.7 MB/s | 320 kB 00:00 2024-03-29T10:48:24.800 INFO:teuthology.orchestra.run.smithi118.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 5.0 MB/s | 298 kB 00:00 2024-03-29T10:48:24.844 INFO:teuthology.orchestra.run.smithi112.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 5.4 MB/s | 742 kB 00:00 2024-03-29T10:48:24.867 INFO:teuthology.orchestra.run.smithi118.stdout:(11/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x8 2.4 MB/s | 5.3 MB 00:02 2024-03-29T10:48:24.892 INFO:teuthology.orchestra.run.smithi118.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.1 MB/s | 99 kB 00:00 2024-03-29T10:48:24.944 INFO:teuthology.orchestra.run.smithi112.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 4.9 MB/s | 498 kB 00:00 2024-03-29T10:48:25.009 INFO:teuthology.orchestra.run.smithi118.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 746 kB/s | 106 kB 00:00 2024-03-29T10:48:25.043 INFO:teuthology.orchestra.run.smithi118.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-29T10:48:25.060 INFO:teuthology.orchestra.run.smithi118.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 876 kB/s | 44 kB 00:00 2024-03-29T10:48:25.160 INFO:teuthology.orchestra.run.smithi118.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 661 kB 00:00 2024-03-29T10:48:25.186 INFO:teuthology.orchestra.run.smithi118.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.2 MB/s | 282 kB 00:00 2024-03-29T10:48:25.219 INFO:teuthology.orchestra.run.smithi118.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 710 kB/s | 41 kB 00:00 2024-03-29T10:48:25.261 INFO:teuthology.orchestra.run.smithi118.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.8 MB/s | 220 kB 00:00 2024-03-29T10:48:25.278 INFO:teuthology.orchestra.run.smithi118.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 592 kB/s | 35 kB 00:00 2024-03-29T10:48:25.320 INFO:teuthology.orchestra.run.smithi118.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 793 kB/s | 46 kB 00:00 2024-03-29T10:48:25.511 INFO:teuthology.orchestra.run.smithi112.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 3.7 MB/s | 9.7 MB 00:02 2024-03-29T10:48:25.520 INFO:teuthology.orchestra.run.smithi118.stdout:(22/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 3.4 MB/s | 9.7 MB 00:02 2024-03-29T10:48:25.578 INFO:teuthology.orchestra.run.smithi112.stdout:(6/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 8.1 MB/s | 5.1 MB 00:00 2024-03-29T10:48:25.603 INFO:teuthology.orchestra.run.smithi112.stdout:(7/32): python3-ceph-argparse-19.0.0-1439.g5dcb 492 kB/s | 45 kB 00:00 2024-03-29T10:48:25.629 INFO:teuthology.orchestra.run.smithi112.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 2.6 MB/s | 131 kB 00:00 2024-03-29T10:48:25.654 INFO:teuthology.orchestra.run.smithi112.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 3.2 MB/s | 162 kB 00:00 2024-03-29T10:48:25.704 INFO:teuthology.orchestra.run.smithi112.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 4.2 MB/s | 320 kB 00:00 2024-03-29T10:48:25.754 INFO:teuthology.orchestra.run.smithi112.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 2.9 MB/s | 298 kB 00:00 2024-03-29T10:48:25.780 INFO:teuthology.orchestra.run.smithi112.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.3 MB/s | 99 kB 00:00 2024-03-29T10:48:25.879 INFO:teuthology.orchestra.run.smithi118.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 73 kB/s | 26 kB 00:00 2024-03-29T10:48:25.963 INFO:teuthology.orchestra.run.smithi118.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 451 kB/s | 309 kB 00:00 2024-03-29T10:48:25.997 INFO:teuthology.orchestra.run.smithi112.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 489 kB/s | 106 kB 00:00 2024-03-29T10:48:26.038 INFO:teuthology.orchestra.run.smithi112.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 616 kB/s | 159 kB 00:00 2024-03-29T10:48:26.046 INFO:teuthology.orchestra.run.smithi118.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 291 kB/s | 49 kB 00:00 2024-03-29T10:48:26.055 INFO:teuthology.orchestra.run.smithi112.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 756 kB/s | 44 kB 00:00 2024-03-29T10:48:26.063 INFO:teuthology.orchestra.run.smithi118.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 672 kB/s | 67 kB 00:00 2024-03-29T10:48:26.188 INFO:teuthology.orchestra.run.smithi118.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-03-29T10:48:26.214 INFO:teuthology.orchestra.run.smithi112.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 661 kB 00:00 2024-03-29T10:48:26.239 INFO:teuthology.orchestra.run.smithi112.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.5 MB/s | 282 kB 00:00 2024-03-29T10:48:26.265 INFO:teuthology.orchestra.run.smithi112.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 830 kB/s | 41 kB 00:00 2024-03-29T10:48:26.307 INFO:teuthology.orchestra.run.smithi112.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2024-03-29T10:48:26.314 INFO:teuthology.orchestra.run.smithi118.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.4 MB/s | 4.4 MB 00:00 2024-03-29T10:48:26.332 INFO:teuthology.orchestra.run.smithi112.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 520 kB/s | 35 kB 00:00 2024-03-29T10:48:26.374 INFO:teuthology.orchestra.run.smithi112.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 693 kB/s | 46 kB 00:00 2024-03-29T10:48:26.431 INFO:teuthology.orchestra.run.smithi118.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.1 MB/s | 838 kB 00:00 2024-03-29T10:48:26.523 INFO:teuthology.orchestra.run.smithi118.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 4.7 MB/s | 1.6 MB 00:00 2024-03-29T10:48:26.574 INFO:teuthology.orchestra.run.smithi112.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-03-29T10:48:26.624 INFO:teuthology.orchestra.run.smithi112.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 522 kB/s | 26 kB 00:00 2024-03-29T10:48:26.683 INFO:teuthology.orchestra.run.smithi112.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 833 kB/s | 49 kB 00:00 2024-03-29T10:48:26.715 INFO:teuthology.orchestra.run.smithi118.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 11 MB/s | 3.1 MB 00:00 2024-03-29T10:48:26.733 INFO:teuthology.orchestra.run.smithi112.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-29T10:48:26.809 INFO:teuthology.orchestra.run.smithi112.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-03-29T10:48:26.868 INFO:teuthology.orchestra.run.smithi112.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.1 MB/s | 838 kB 00:00 2024-03-29T10:48:26.893 INFO:teuthology.orchestra.run.smithi112.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-29T10:48:26.993 INFO:teuthology.orchestra.run.smithi112.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-03-29T10:48:27.318 INFO:teuthology.orchestra.run.smithi112.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 7.9 MB/s | 3.4 MB 00:00 2024-03-29T10:48:27.493 INFO:teuthology.orchestra.run.smithi112.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 6.3 MB/s | 3.1 MB 00:00 2024-03-29T10:48:27.556 INFO:teuthology.orchestra.run.smithi118.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 2.7 MB/s | 3.4 MB 00:01 2024-03-29T10:48:27.564 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:48:27.564 INFO:teuthology.orchestra.run.smithi118.stdout:Total 11 MB/s | 58 MB 00:05 2024-03-29T10:48:27.564 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-29T10:48:27.601 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-29T10:48:27.601 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-29T10:48:28.082 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-29T10:48:28.082 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-29T10:48:29.352 INFO:teuthology.orchestra.run.smithi112.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 3.1 MB/s | 20 MB 00:06 2024-03-29T10:48:29.356 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:48:29.356 INFO:teuthology.orchestra.run.smithi112.stdout:Total 8.8 MB/s | 58 MB 00:06 2024-03-29T10:48:29.357 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-03-29T10:48:29.394 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-03-29T10:48:29.395 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-03-29T10:48:29.565 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-29T10:48:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:48:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:48:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:48:29.743 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:48:29.759 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:48:29.869 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-03-29T10:48:29.869 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-03-29T10:48:29.927 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:48:29.982 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:48:30.315 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:48:30.389 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:48:30.464 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:48:30.553 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:48:30.582 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:48:30.628 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:48:30.657 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:48:30.701 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:48:30.778 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:48:30.801 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:48:30.865 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:48:30.892 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:48:31.002 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:48:31.034 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:48:31.075 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:48:31.118 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:48:31.153 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:48:31.186 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:48:31.338 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-03-29T10:48:31.424 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:48:31.465 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:48:31.490 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:48:31.494 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:48:31.513 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:48:31.524 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:48:31.529 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:48:31.596 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:48:31.629 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:48:31.675 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:48:31.739 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:48:31.793 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:48:32.123 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:48:32.128 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:48:32.152 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:48:32.175 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:48:32.224 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:48:32.259 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:48:32.291 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:48:32.330 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:48:32.349 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:48:32.377 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:48:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:48:32.454 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:48:32.496 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:48:32.565 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:48:32.588 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:48:32.643 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:48:32.678 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:48:32.788 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:48:32.829 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:48:32.869 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:48:32.902 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:48:32.931 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:48:32.972 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:48:33.292 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:48:33.327 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:48:33.408 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:48:33.440 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:48:33.483 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:48:33.940 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:48:33.971 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:48:34.042 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:48:34.111 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:48:34.140 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:48:34.483 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:48:34.544 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:48:35.087 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:48:35.172 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:48:35.269 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:48:35.269 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:48:35.269 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:48:35.269 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:48:35.307 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:48:36.353 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:48:36.403 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:48:36.955 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:48:37.115 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:48:37.188 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:48:37.189 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:48:37.189 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:48:37.189 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:48:37.216 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:48:45.837 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:48:45.837 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /sys 2024-03-29T10:48:45.838 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /proc 2024-03-29T10:48:45.838 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /mnt 2024-03-29T10:48:45.838 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /var/tmp 2024-03-29T10:48:45.838 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /home 2024-03-29T10:48:45.838 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /root 2024-03-29T10:48:45.838 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /tmp 2024-03-29T10:48:45.838 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:48:46.800 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:48:46.803 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:48:46.804 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /sys 2024-03-29T10:48:46.804 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /proc 2024-03-29T10:48:46.804 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /mnt 2024-03-29T10:48:46.804 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /var/tmp 2024-03-29T10:48:46.804 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /home 2024-03-29T10:48:46.804 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /root 2024-03-29T10:48:46.804 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /tmp 2024-03-29T10:48:46.804 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:48:46.825 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:48:46.825 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:48:46.826 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:48:46.826 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:48:46.826 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:48:46.826 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:48:46.828 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:48:46.920 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:48:46.920 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:48:47.758 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:48:47.782 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:48:47.782 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:48:47.782 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:48:47.783 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:48:47.783 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:48:47.783 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:48:47.784 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:48:47.853 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:48:47.853 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:48:48.050 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:48:48.051 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:48:48.051 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:48:48.051 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:48:48.051 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:48:48.051 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:48:48.051 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:48:48.051 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:48:48.051 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:48:48.051 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:48:48.051 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:48:48.051 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:48:48.051 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:48:48.051 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:48:48.052 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:48:48.052 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:48:48.052 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:48:48.052 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:48:48.052 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:48:48.052 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:48:48.052 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:48:48.052 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:48:48.052 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:48:48.052 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:48:48.052 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:48:48.052 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:48:48.052 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:48:48.053 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:48:48.054 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:48:48.054 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:48:48.054 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:48:48.054 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:48:48.054 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:48:48.054 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:48:48.972 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:48:48.972 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:48:48.973 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:48:48.973 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:48:48.973 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:48:48.973 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:48:48.973 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:48:48.973 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:48:48.973 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:48:48.973 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:48:48.973 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:48:48.973 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:48:48.973 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:48:48.973 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:48:48.973 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:48:48.973 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:48:48.974 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:48:48.974 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:48:48.974 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:48:48.974 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:48:48.974 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:48:48.974 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:48:48.974 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:48:48.974 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:48:48.974 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:48:48.974 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:48:48.974 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:48:48.974 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:48:48.974 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:48:48.974 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:48:48.975 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:48:48.975 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:48:48.975 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:48:48.975 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:48:49.078 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:48:49.078 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:48:49.078 INFO:teuthology.orchestra.run.smithi118.stdout:Upgraded: 2024-03-29T10:48:49.078 INFO:teuthology.orchestra.run.smithi118.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:49.078 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:49.078 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-29T10:48:49.078 INFO:teuthology.orchestra.run.smithi118.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:48:49.078 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:49.078 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:49.078 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:49.079 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:49.079 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:48:49.079 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:48:49.079 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:48:49.079 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:49.079 INFO:teuthology.orchestra.run.smithi118.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:48:49.079 INFO:teuthology.orchestra.run.smithi118.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:48:49.079 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:48:49.079 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:49.079 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:48:49.079 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:49.079 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:48:49.079 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:48:49.079 INFO:teuthology.orchestra.run.smithi118.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:48:49.080 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:49.080 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:49.080 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:49.080 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:48:49.080 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:49.080 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:49.080 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:49.080 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:48:49.080 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:48:49.080 INFO:teuthology.orchestra.run.smithi118.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:48:49.080 INFO:teuthology.orchestra.run.smithi118.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:48:49.080 INFO:teuthology.orchestra.run.smithi118.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:48:49.080 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:48:49.080 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:48:49.432 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-test 2024-03-29T10:48:49.999 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:31 ago on Fri 29 Mar 2024 10:48:19 AM UTC. 2024-03-29T10:48:50.087 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:48:50.087 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:48:50.088 INFO:teuthology.orchestra.run.smithi112.stdout:Upgraded: 2024-03-29T10:48:50.088 INFO:teuthology.orchestra.run.smithi112.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:50.088 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:50.088 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-03-29T10:48:50.088 INFO:teuthology.orchestra.run.smithi112.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:48:50.088 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:50.088 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:50.088 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:50.088 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:50.089 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:48:50.089 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:48:50.089 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:48:50.089 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:50.089 INFO:teuthology.orchestra.run.smithi112.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:48:50.089 INFO:teuthology.orchestra.run.smithi112.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:48:50.089 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:48:50.089 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:50.089 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:48:50.089 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:50.090 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:48:50.090 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:48:50.090 INFO:teuthology.orchestra.run.smithi112.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:48:50.090 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:50.090 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:50.090 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:50.090 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:48:50.090 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:50.090 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:50.091 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:48:50.091 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:48:50.091 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:48:50.091 INFO:teuthology.orchestra.run.smithi112.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:48:50.091 INFO:teuthology.orchestra.run.smithi112.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:48:50.091 INFO:teuthology.orchestra.run.smithi112.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:48:50.091 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:48:50.091 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:48:50.153 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:48:50.154 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:48:50.154 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-29T10:48:50.154 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:48:50.154 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-29T10:48:50.154 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:48:50.154 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-29T10:48:50.154 INFO:teuthology.orchestra.run.smithi118.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:48:50.154 INFO:teuthology.orchestra.run.smithi118.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:48:50.154 INFO:teuthology.orchestra.run.smithi118.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:48:50.154 INFO:teuthology.orchestra.run.smithi118.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:48:50.154 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:48:50.155 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:48:50.155 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-29T10:48:50.155 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:48:50.155 INFO:teuthology.orchestra.run.smithi118.stdout:Install 6 Packages 2024-03-29T10:48:50.155 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:48:50.156 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 48 M 2024-03-29T10:48:50.156 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 204 M 2024-03-29T10:48:50.156 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-29T10:48:50.448 INFO:teuthology.orchestra.run.smithi118.stdout:(1/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 637 kB/s | 163 kB 00:00 2024-03-29T10:48:50.471 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-test 2024-03-29T10:48:50.507 INFO:teuthology.orchestra.run.smithi118.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 689 kB/s | 217 kB 00:00 2024-03-29T10:48:50.607 INFO:teuthology.orchestra.run.smithi118.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.8 MB/s | 300 kB 00:00 2024-03-29T10:48:50.624 INFO:teuthology.orchestra.run.smithi118.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 535 kB/s | 63 kB 00:00 2024-03-29T10:48:50.750 INFO:teuthology.orchestra.run.smithi118.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-29T10:48:51.028 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:31 ago on Fri 29 Mar 2024 10:48:20 AM UTC. 2024-03-29T10:48:51.183 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:48:51.183 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:48:51.183 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-03-29T10:48:51.183 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:48:51.183 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-03-29T10:48:51.184 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:48:51.184 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-03-29T10:48:51.184 INFO:teuthology.orchestra.run.smithi112.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:48:51.184 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:48:51.184 INFO:teuthology.orchestra.run.smithi112.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:48:51.184 INFO:teuthology.orchestra.run.smithi112.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:48:51.184 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:48:51.184 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:48:51.184 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-03-29T10:48:51.184 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:48:51.184 INFO:teuthology.orchestra.run.smithi112.stdout:Install 6 Packages 2024-03-29T10:48:51.184 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:48:51.185 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 48 M 2024-03-29T10:48:51.185 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 204 M 2024-03-29T10:48:51.185 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-03-29T10:48:51.475 INFO:teuthology.orchestra.run.smithi112.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-29T10:48:51.542 INFO:teuthology.orchestra.run.smithi112.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 300 kB 00:00 2024-03-29T10:48:51.575 INFO:teuthology.orchestra.run.smithi112.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T10:48:51.617 INFO:teuthology.orchestra.run.smithi112.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 518 kB/s | 163 kB 00:00 2024-03-29T10:48:52.101 INFO:teuthology.orchestra.run.smithi112.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 356 kB/s | 187 kB 00:00 2024-03-29T10:48:53.050 INFO:teuthology.orchestra.run.smithi118.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 17 MB/s | 47 MB 00:02 2024-03-29T10:48:53.053 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:48:53.053 INFO:teuthology.orchestra.run.smithi118.stdout:Total 17 MB/s | 48 MB 00:02 2024-03-29T10:48:53.067 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-29T10:48:53.087 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-29T10:48:53.088 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-29T10:48:53.471 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-29T10:48:53.471 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-29T10:48:54.045 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-29T10:48:54.125 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:48:54.169 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:48:54.209 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:48:54.251 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:48:54.278 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:48:58.763 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:48:58.836 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:49:01.756 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:49:01.756 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:49:01.756 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:49:01.756 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:49:01.756 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:49:01.756 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:49:02.141 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:49:02.141 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:02.141 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-29T10:49:02.141 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:02.141 INFO:teuthology.orchestra.run.smithi118.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:49:02.142 INFO:teuthology.orchestra.run.smithi118.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:02.142 INFO:teuthology.orchestra.run.smithi118.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:49:02.142 INFO:teuthology.orchestra.run.smithi118.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:49:02.142 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:49:02.142 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:02.142 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:49:02.446 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph 2024-03-29T10:49:03.020 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:44 ago on Fri 29 Mar 2024 10:48:19 AM UTC. 2024-03-29T10:49:03.175 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:49:03.177 INFO:teuthology.orchestra.run.smithi118.stdout:========================================================================================= 2024-03-29T10:49:03.177 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-29T10:49:03.178 INFO:teuthology.orchestra.run.smithi118.stdout:========================================================================================= 2024-03-29T10:49:03.178 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-29T10:49:03.178 INFO:teuthology.orchestra.run.smithi118.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:49:03.178 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-29T10:49:03.178 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:49:03.178 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:49:03.178 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:49:03.178 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:49:03.178 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:49:03.178 INFO:teuthology.orchestra.run.smithi118.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:49:03.178 INFO:teuthology.orchestra.run.smithi118.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:49:03.179 INFO:teuthology.orchestra.run.smithi118.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:49:03.179 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:49:03.179 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:49:03.179 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:49:03.179 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:49:03.179 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:49:03.179 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:49:03.179 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:49:03.179 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:49:03.179 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:49:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:49:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:49:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:49:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:49:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:49:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:49:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:49:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:49:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:49:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:49:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:49:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:49:03.181 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:49:03.181 INFO:teuthology.orchestra.run.smithi118.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:49:03.181 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:49:03.181 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:49:03.181 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:49:03.181 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:49:03.181 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:03.181 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-29T10:49:03.181 INFO:teuthology.orchestra.run.smithi118.stdout:========================================================================================= 2024-03-29T10:49:03.181 INFO:teuthology.orchestra.run.smithi118.stdout:Install 36 Packages 2024-03-29T10:49:03.181 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:03.182 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 30 M 2024-03-29T10:49:03.182 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 107 M 2024-03-29T10:49:03.182 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-29T10:49:03.988 INFO:teuthology.orchestra.run.smithi118.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 37 kB/s | 6.6 kB 00:00 2024-03-29T10:49:04.855 INFO:teuthology.orchestra.run.smithi118.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.0 MB/s | 2.1 MB 00:01 2024-03-29T10:49:04.889 INFO:teuthology.orchestra.run.smithi118.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.4 MB/s | 1.5 MB 00:01 2024-03-29T10:49:04.964 INFO:teuthology.orchestra.run.smithi118.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 3.2 MB/s | 247 kB 00:00 2024-03-29T10:49:05.098 INFO:teuthology.orchestra.run.smithi118.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-29T10:49:05.140 INFO:teuthology.orchestra.run.smithi118.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T10:49:05.173 INFO:teuthology.orchestra.run.smithi118.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 947 kB/s | 32 kB 00:00 2024-03-29T10:49:05.207 INFO:teuthology.orchestra.run.smithi118.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T10:49:05.241 INFO:teuthology.orchestra.run.smithi118.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T10:49:05.283 INFO:teuthology.orchestra.run.smithi118.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-29T10:49:05.358 INFO:teuthology.orchestra.run.smithi118.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T10:49:05.391 INFO:teuthology.orchestra.run.smithi118.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T10:49:05.425 INFO:teuthology.orchestra.run.smithi118.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T10:49:05.467 INFO:teuthology.orchestra.run.smithi118.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-29T10:49:05.543 INFO:teuthology.orchestra.run.smithi118.stdout:(15/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 3.0 MB/s | 4.7 MB 00:01 2024-03-29T10:49:05.710 INFO:teuthology.orchestra.run.smithi118.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 134 kB/s | 22 kB 00:00 2024-03-29T10:49:05.976 INFO:teuthology.orchestra.run.smithi118.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 623 kB/s | 151 kB 00:00 2024-03-29T10:49:06.035 INFO:teuthology.orchestra.run.smithi118.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 744 kB/s | 43 kB 00:00 2024-03-29T10:49:06.119 INFO:teuthology.orchestra.run.smithi118.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-03-29T10:49:06.303 INFO:teuthology.orchestra.run.smithi118.stdout:(20/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 12 MB/s | 17 MB 00:01 2024-03-29T10:49:06.328 INFO:teuthology.orchestra.run.smithi118.stdout:(21/36): python3-requests-2.25.1-8.el9.noarch.r 145 kB/s | 125 kB 00:00 2024-03-29T10:49:06.353 INFO:teuthology.orchestra.run.smithi118.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-03-29T10:49:06.420 INFO:teuthology.orchestra.run.smithi118.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 91 kB/s | 11 kB 00:00 2024-03-29T10:49:06.445 INFO:teuthology.orchestra.run.smithi118.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 252 kB/s | 23 kB 00:00 2024-03-29T10:49:06.479 INFO:teuthology.orchestra.run.smithi118.stdout:(25/36): python3-jaraco-classes-3.2.1-5.el9.noa 117 kB/s | 18 kB 00:00 2024-03-29T10:49:06.504 INFO:teuthology.orchestra.run.smithi118.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 232 kB/s | 19 kB 00:00 2024-03-29T10:49:06.529 INFO:teuthology.orchestra.run.smithi118.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 234 kB/s | 20 kB 00:00 2024-03-29T10:49:06.563 INFO:teuthology.orchestra.run.smithi118.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-29T10:49:06.605 INFO:teuthology.orchestra.run.smithi118.stdout:(29/36): python3-logutils-0.3.5-21.el9.noarch.r 367 kB/s | 46 kB 00:00 2024-03-29T10:49:06.630 INFO:teuthology.orchestra.run.smithi118.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 244 kB/s | 16 kB 00:00 2024-03-29T10:49:06.681 INFO:teuthology.orchestra.run.smithi118.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-03-29T10:49:06.706 INFO:teuthology.orchestra.run.smithi118.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-03-29T10:49:06.731 INFO:teuthology.orchestra.run.smithi118.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 713 kB/s | 90 kB 00:00 2024-03-29T10:49:06.757 INFO:teuthology.orchestra.run.smithi118.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-29T10:49:06.807 INFO:teuthology.orchestra.run.smithi118.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 266 kB/s | 20 kB 00:00 2024-03-29T10:49:06.840 INFO:teuthology.orchestra.run.smithi118.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-03-29T10:49:06.843 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:06.844 INFO:teuthology.orchestra.run.smithi118.stdout:Total 8.1 MB/s | 30 MB 00:03 2024-03-29T10:49:06.867 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-29T10:49:06.907 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-29T10:49:06.908 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-29T10:49:07.314 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-29T10:49:07.315 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-29T10:49:08.662 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-29T10:49:08.726 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:49:08.762 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:49:08.804 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:49:08.875 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:49:08.936 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:49:08.969 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:49:09.060 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:49:09.138 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:49:09.180 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:49:09.252 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:49:09.306 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:49:09.344 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:49:09.393 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:49:09.448 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:49:09.634 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:49:09.682 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:49:09.721 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:49:09.756 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:49:09.785 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:49:09.863 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:49:09.896 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:49:09.929 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:49:09.969 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:49:10.005 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:49:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:49:10.130 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:49:10.170 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:49:10.243 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:49:10.381 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:49:10.507 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:49:10.599 INFO:teuthology.orchestra.run.smithi112.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 2.4 MB/s | 47 MB 00:19 2024-03-29T10:49:10.602 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:10.602 INFO:teuthology.orchestra.run.smithi112.stdout:Total 2.5 MB/s | 48 MB 00:19 2024-03-29T10:49:10.617 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-03-29T10:49:10.638 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-03-29T10:49:10.638 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-03-29T10:49:10.662 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:49:10.684 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:49:10.709 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:49:10.710 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:49:10.710 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:49:10.710 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:49:10.710 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:49:10.710 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:11.018 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-03-29T10:49:11.018 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-03-29T10:49:11.591 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-03-29T10:49:11.666 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:49:11.703 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:49:11.735 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:49:11.782 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:49:11.802 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:49:12.177 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:49:12.203 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:49:12.204 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:49:12.204 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:49:12.204 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:49:12.204 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:49:12.204 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:12.644 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:49:12.667 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:49:12.668 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:49:12.668 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:49:12.668 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:49:12.669 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:49:12.669 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:12.870 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:49:12.893 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:49:12.893 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:49:12.893 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:49:12.893 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:49:12.893 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:49:12.894 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:12.960 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:49:15.166 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:49:15.167 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:49:15.167 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:49:15.167 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:49:15.167 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:49:15.167 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:49:15.167 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:49:15.167 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:49:15.167 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:49:15.167 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:49:15.167 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:49:15.167 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:49:15.168 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:49:15.168 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:49:15.168 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:49:15.168 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:49:15.168 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:49:15.168 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:49:15.168 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:49:15.168 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:49:15.168 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:49:15.168 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:49:15.168 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:49:15.168 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:49:15.168 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:49:15.169 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:49:15.169 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:49:15.169 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:49:15.169 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:49:15.169 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:49:15.169 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:49:15.169 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:49:15.169 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:49:15.169 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:49:15.169 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:49:15.169 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:49:16.207 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:49:16.207 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:16.207 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-29T10:49:16.208 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:16.208 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:16.208 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:16.208 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:49:16.208 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:16.208 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:16.208 INFO:teuthology.orchestra.run.smithi118.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:49:16.208 INFO:teuthology.orchestra.run.smithi118.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:49:16.208 INFO:teuthology.orchestra.run.smithi118.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:49:16.208 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:49:16.208 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:49:16.208 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:49:16.208 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:49:16.208 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:49:16.209 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:49:16.209 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:49:16.209 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:49:16.209 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:49:16.209 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:49:16.209 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:49:16.209 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:49:16.209 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:49:16.209 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:49:16.209 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:49:16.209 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:49:16.209 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:49:16.209 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:49:16.209 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:49:16.210 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:49:16.210 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:49:16.210 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:49:16.210 INFO:teuthology.orchestra.run.smithi118.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:49:16.210 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:49:16.210 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:49:16.210 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:49:16.210 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:49:16.211 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:16.211 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:49:16.271 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:49:16.351 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:49:16.556 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-base 2024-03-29T10:49:17.125 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:58 ago on Fri 29 Mar 2024 10:48:19 AM UTC. 2024-03-29T10:49:17.224 INFO:teuthology.orchestra.run.smithi118.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:49:17.275 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:49:17.276 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-29T10:49:17.276 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:49:17.347 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install cephadm 2024-03-29T10:49:17.911 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:58 ago on Fri 29 Mar 2024 10:48:19 AM UTC. 2024-03-29T10:49:18.062 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:49:18.063 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:49:18.063 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-29T10:49:18.063 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:49:18.063 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-29T10:49:18.063 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:49:18.063 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:18.064 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-29T10:49:18.064 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:49:18.064 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-03-29T10:49:18.064 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:18.064 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 747 k 2024-03-29T10:49:18.064 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 754 k 2024-03-29T10:49:18.064 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-29T10:49:18.342 INFO:teuthology.orchestra.run.smithi118.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.6 MB/s | 747 kB 00:00 2024-03-29T10:49:18.343 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:18.343 INFO:teuthology.orchestra.run.smithi118.stdout:Total 2.6 MB/s | 747 kB 00:00 2024-03-29T10:49:18.343 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-29T10:49:18.348 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-29T10:49:18.348 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-29T10:49:18.354 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-29T10:49:18.354 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-29T10:49:18.426 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-29T10:49:18.681 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:49:18.681 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:49:18.681 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:49:18.681 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:49:18.681 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:49:18.681 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:49:18.766 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:49:18.847 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:49:19.064 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:49:19.065 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:19.065 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-03-29T10:49:19.065 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:19.065 INFO:teuthology.orchestra.run.smithi112.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:49:19.065 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:19.065 INFO:teuthology.orchestra.run.smithi112.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:49:19.065 INFO:teuthology.orchestra.run.smithi112.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:49:19.065 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:49:19.065 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:19.065 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:49:19.235 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:49:19.401 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph 2024-03-29T10:49:19.519 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:49:19.519 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:19.519 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-29T10:49:19.519 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:49:19.519 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:19.519 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:49:19.736 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:49:19.968 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 10:48:20 AM UTC. 2024-03-29T10:49:20.124 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:49:20.126 INFO:teuthology.orchestra.run.smithi112.stdout:========================================================================================= 2024-03-29T10:49:20.127 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-03-29T10:49:20.127 INFO:teuthology.orchestra.run.smithi112.stdout:========================================================================================= 2024-03-29T10:49:20.127 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-03-29T10:49:20.127 INFO:teuthology.orchestra.run.smithi112.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:49:20.127 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-03-29T10:49:20.127 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:49:20.127 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:49:20.127 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:49:20.127 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:49:20.127 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:49:20.127 INFO:teuthology.orchestra.run.smithi112.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:49:20.128 INFO:teuthology.orchestra.run.smithi112.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:49:20.128 INFO:teuthology.orchestra.run.smithi112.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:49:20.128 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:49:20.128 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:49:20.128 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:49:20.128 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:49:20.128 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:49:20.128 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:49:20.128 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:49:20.128 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:49:20.128 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:49:20.129 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:49:20.129 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:49:20.129 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:49:20.129 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:49:20.129 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:49:20.129 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:49:20.129 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:49:20.129 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:49:20.129 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:49:20.129 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:49:20.129 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:49:20.129 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:49:20.130 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:49:20.130 INFO:teuthology.orchestra.run.smithi112.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:49:20.130 INFO:teuthology.orchestra.run.smithi112.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:49:20.130 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:49:20.130 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:49:20.130 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:49:20.130 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:20.130 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-03-29T10:49:20.130 INFO:teuthology.orchestra.run.smithi112.stdout:========================================================================================= 2024-03-29T10:49:20.130 INFO:teuthology.orchestra.run.smithi112.stdout:Install 36 Packages 2024-03-29T10:49:20.130 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:20.131 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 30 M 2024-03-29T10:49:20.131 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 107 M 2024-03-29T10:49:20.131 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-03-29T10:49:20.305 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 10:48:19 AM UTC. 2024-03-29T10:49:20.461 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:49:20.462 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:49:20.462 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2024-03-29T10:49:20.462 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:49:20.462 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-29T10:49:20.462 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:49:20.462 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:20.462 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-29T10:49:20.462 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:49:20.462 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-03-29T10:49:20.462 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:20.462 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 145 k 2024-03-29T10:49:20.462 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 436 k 2024-03-29T10:49:20.463 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-29T10:49:20.588 INFO:teuthology.orchestra.run.smithi112.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 20 kB/s | 6.6 kB 00:00 2024-03-29T10:49:20.950 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 298 kB/s | 145 kB 00:00 2024-03-29T10:49:20.950 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:20.950 INFO:teuthology.orchestra.run.smithi118.stdout:Total 297 kB/s | 145 kB 00:00 2024-03-29T10:49:20.951 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-29T10:49:20.958 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-29T10:49:20.958 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-29T10:49:21.011 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-29T10:49:21.011 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-29T10:49:21.146 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-29T10:49:21.209 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:49:21.234 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:49:21.234 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:49:21.234 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:49:21.234 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:21.296 INFO:teuthology.orchestra.run.smithi112.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.1 MB/s | 2.1 MB 00:01 2024-03-29T10:49:21.430 INFO:teuthology.orchestra.run.smithi112.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.3 MB/s | 1.5 MB 00:01 2024-03-29T10:49:21.580 INFO:teuthology.orchestra.run.smithi112.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.6 MB/s | 247 kB 00:00 2024-03-29T10:49:21.697 INFO:teuthology.orchestra.run.smithi112.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-29T10:49:21.739 INFO:teuthology.orchestra.run.smithi112.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T10:49:21.764 INFO:teuthology.orchestra.run.smithi112.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-29T10:49:21.790 INFO:teuthology.orchestra.run.smithi112.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2024-03-29T10:49:21.815 INFO:teuthology.orchestra.run.smithi112.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-29T10:49:21.857 INFO:teuthology.orchestra.run.smithi112.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-29T10:49:21.924 INFO:teuthology.orchestra.run.smithi112.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T10:49:21.957 INFO:teuthology.orchestra.run.smithi112.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T10:49:21.991 INFO:teuthology.orchestra.run.smithi112.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T10:49:22.025 INFO:teuthology.orchestra.run.smithi112.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-29T10:49:22.219 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:49:22.219 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:22.219 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-29T10:49:22.219 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:22.220 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:22.220 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:49:22.442 INFO:teuthology.orchestra.run.smithi112.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 300 kB/s | 125 kB 00:00 2024-03-29T10:49:22.462 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr 2024-03-29T10:49:22.542 INFO:teuthology.orchestra.run.smithi112.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-03-29T10:49:22.659 INFO:teuthology.orchestra.run.smithi112.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-03-29T10:49:22.693 INFO:teuthology.orchestra.run.smithi112.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-03-29T10:49:22.768 INFO:teuthology.orchestra.run.smithi112.stdout:(19/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 2.1 MB/s | 4.7 MB 00:02 2024-03-29T10:49:22.793 INFO:teuthology.orchestra.run.smithi112.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-03-29T10:49:22.827 INFO:teuthology.orchestra.run.smithi112.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 324 kB/s | 11 kB 00:00 2024-03-29T10:49:22.860 INFO:teuthology.orchestra.run.smithi112.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 530 kB/s | 18 kB 00:00 2024-03-29T10:49:22.886 INFO:teuthology.orchestra.run.smithi112.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2024-03-29T10:49:22.911 INFO:teuthology.orchestra.run.smithi112.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-03-29T10:49:22.936 INFO:teuthology.orchestra.run.smithi112.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-03-29T10:49:22.962 INFO:teuthology.orchestra.run.smithi112.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-03-29T10:49:22.987 INFO:teuthology.orchestra.run.smithi112.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 926 kB/s | 46 kB 00:00 2024-03-29T10:49:23.012 INFO:teuthology.orchestra.run.smithi112.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-03-29T10:49:23.028 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:04 ago on Fri 29 Mar 2024 10:48:19 AM UTC. 2024-03-29T10:49:23.046 INFO:teuthology.orchestra.run.smithi112.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-03-29T10:49:23.072 INFO:teuthology.orchestra.run.smithi112.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-03-29T10:49:23.097 INFO:teuthology.orchestra.run.smithi112.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-03-29T10:49:23.122 INFO:teuthology.orchestra.run.smithi112.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-03-29T10:49:23.130 INFO:teuthology.orchestra.run.smithi118.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:49:23.147 INFO:teuthology.orchestra.run.smithi112.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-29T10:49:23.181 INFO:teuthology.orchestra.run.smithi112.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.1 MB/s | 427 kB 00:00 2024-03-29T10:49:23.186 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:49:23.187 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-29T10:49:23.188 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:49:23.197 INFO:teuthology.orchestra.run.smithi112.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-03-29T10:49:23.250 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:49:23.381 INFO:teuthology.orchestra.run.smithi112.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 8.1 MB/s | 17 MB 00:02 2024-03-29T10:49:23.385 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:23.385 INFO:teuthology.orchestra.run.smithi112.stdout:Total 9.1 MB/s | 30 MB 00:03 2024-03-29T10:49:23.410 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-03-29T10:49:23.451 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-03-29T10:49:23.451 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-03-29T10:49:23.808 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:04 ago on Fri 29 Mar 2024 10:48:19 AM UTC. 2024-03-29T10:49:23.838 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-03-29T10:49:23.839 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-03-29T10:49:23.958 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:49:23.959 INFO:teuthology.orchestra.run.smithi118.stdout:====================================================================================== 2024-03-29T10:49:23.959 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-29T10:49:23.959 INFO:teuthology.orchestra.run.smithi118.stdout:====================================================================================== 2024-03-29T10:49:23.959 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-29T10:49:23.960 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:49:23.960 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-29T10:49:23.960 INFO:teuthology.orchestra.run.smithi118.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:49:23.960 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:49:23.960 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:49:23.960 INFO:teuthology.orchestra.run.smithi118.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:49:23.960 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:49:23.960 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:49:23.960 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:49:23.960 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:49:23.960 INFO:teuthology.orchestra.run.smithi118.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:49:23.960 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:49:23.960 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:49:23.961 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:23.961 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-29T10:49:23.961 INFO:teuthology.orchestra.run.smithi118.stdout:====================================================================================== 2024-03-29T10:49:23.961 INFO:teuthology.orchestra.run.smithi118.stdout:Install 12 Packages 2024-03-29T10:49:23.961 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:23.961 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 6.9 M 2024-03-29T10:49:23.961 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 26 M 2024-03-29T10:49:23.962 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-29T10:49:24.755 INFO:teuthology.orchestra.run.smithi118.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 41 kB/s | 15 kB 00:00 2024-03-29T10:49:24.855 INFO:teuthology.orchestra.run.smithi118.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 66 kB/s | 31 kB 00:00 2024-03-29T10:49:24.972 INFO:teuthology.orchestra.run.smithi118.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 238 kB 00:00 2024-03-29T10:49:25.005 INFO:teuthology.orchestra.run.smithi118.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.0 MB/s | 1.0 MB 00:00 2024-03-29T10:49:25.181 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-03-29T10:49:25.197 INFO:teuthology.orchestra.run.smithi118.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.8 MB/s | 863 kB 00:00 2024-03-29T10:49:25.225 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:49:25.267 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:49:25.294 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:49:25.331 INFO:teuthology.orchestra.run.smithi118.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 146 kB/s | 19 kB 00:00 2024-03-29T10:49:25.363 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:49:25.365 INFO:teuthology.orchestra.run.smithi118.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.5 MB/s | 548 kB 00:00 2024-03-29T10:49:25.401 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:49:25.434 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:49:25.449 INFO:teuthology.orchestra.run.smithi118.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.7 MB/s | 144 kB 00:00 2024-03-29T10:49:25.490 INFO:teuthology.orchestra.run.smithi118.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 743 kB/s | 31 kB 00:00 2024-03-29T10:49:25.524 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:49:25.574 INFO:teuthology.orchestra.run.smithi118.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-29T10:49:25.603 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:49:25.644 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:49:25.716 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:49:25.750 INFO:teuthology.orchestra.run.smithi118.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.9 MB/s | 2.0 MB 00:00 2024-03-29T10:49:25.786 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:49:25.799 INFO:teuthology.orchestra.run.smithi118.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 1.3 MB/s | 1.8 MB 00:01 2024-03-29T10:49:25.802 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:25.802 INFO:teuthology.orchestra.run.smithi118.stdout:Total 3.7 MB/s | 6.9 MB 00:01 2024-03-29T10:49:25.820 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-29T10:49:25.825 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:49:25.836 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-29T10:49:25.836 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-29T10:49:25.873 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:49:25.929 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:49:26.025 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-29T10:49:26.025 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-29T10:49:26.111 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:49:26.155 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:49:26.184 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:49:26.219 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:49:26.249 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:49:26.324 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:49:26.360 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:49:26.401 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:49:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:49:26.476 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:49:26.519 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:49:26.569 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-29T10:49:26.599 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:49:26.633 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:49:26.650 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:49:26.693 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:49:26.705 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:49:26.740 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:49:26.779 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:49:26.839 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:49:26.888 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:49:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:49:27.013 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:49:27.107 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:49:27.116 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:49:27.138 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:49:27.151 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:49:27.163 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:49:27.163 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:49:27.163 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:49:27.163 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:49:27.163 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:49:27.164 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:27.179 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:49:27.211 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:49:27.455 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:49:27.495 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:49:28.494 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:49:28.494 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:49:28.494 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:49:28.495 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:49:28.495 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:49:28.495 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:49:28.495 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:49:28.495 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:49:28.495 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:49:28.495 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:49:28.495 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:49:28.495 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:49:28.606 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:49:28.633 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:49:28.633 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:49:28.633 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:49:28.633 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:49:28.633 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:49:28.633 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:28.850 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:49:28.850 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:28.850 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-29T10:49:28.851 INFO:teuthology.orchestra.run.smithi118.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:49:28.851 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:49:28.851 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:49:28.851 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:49:28.851 INFO:teuthology.orchestra.run.smithi118.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:49:28.851 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:49:28.851 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:49:28.851 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:49:28.851 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:49:28.851 INFO:teuthology.orchestra.run.smithi118.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:49:28.851 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:49:28.851 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:49:28.852 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:28.852 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:49:29.082 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:49:29.103 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:49:29.105 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:49:29.105 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:49:29.105 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:49:29.105 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:49:29.105 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:49:29.105 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:29.307 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:49:29.330 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:49:29.330 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:49:29.331 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:49:29.331 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:49:29.331 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:49:29.331 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:29.396 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:49:29.661 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:10 ago on Fri 29 Mar 2024 10:48:19 AM UTC. 2024-03-29T10:49:29.812 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:49:29.813 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================================ 2024-03-29T10:49:29.813 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-29T10:49:29.813 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================================ 2024-03-29T10:49:29.813 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-29T10:49:29.813 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:49:29.813 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-29T10:49:29.813 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:49:29.813 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:49:29.814 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:49:29.814 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:49:29.814 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:49:29.814 INFO:teuthology.orchestra.run.smithi118.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:49:29.814 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:49:29.814 INFO:teuthology.orchestra.run.smithi118.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:49:29.814 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:49:29.814 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:49:29.814 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:49:29.815 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:29.815 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-29T10:49:29.815 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================================ 2024-03-29T10:49:29.815 INFO:teuthology.orchestra.run.smithi118.stdout:Install 12 Packages 2024-03-29T10:49:29.815 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:29.815 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 37 M 2024-03-29T10:49:29.815 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 209 M 2024-03-29T10:49:29.816 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-29T10:49:30.199 INFO:teuthology.orchestra.run.smithi118.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 211 kB/s | 32 kB 00:00 2024-03-29T10:49:30.232 INFO:teuthology.orchestra.run.smithi118.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 516 kB/s | 17 kB 00:00 2024-03-29T10:49:30.274 INFO:teuthology.orchestra.run.smithi118.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 821 kB/s | 34 kB 00:00 2024-03-29T10:49:30.525 INFO:teuthology.orchestra.run.smithi118.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 19 MB/s | 4.7 MB 00:00 2024-03-29T10:49:30.617 INFO:teuthology.orchestra.run.smithi118.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.3 MB/s | 3.0 MB 00:00 2024-03-29T10:49:30.710 INFO:teuthology.orchestra.run.smithi118.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 26 MB/s | 4.8 MB 00:00 2024-03-29T10:49:30.735 INFO:teuthology.orchestra.run.smithi118.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.9 MB/s | 217 kB 00:00 2024-03-29T10:49:31.063 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:49:31.063 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:49:31.063 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:49:31.063 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:49:31.063 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:49:31.063 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:49:31.063 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:49:31.063 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:49:31.063 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:49:31.064 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:49:31.064 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:49:31.064 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:49:31.064 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:49:31.064 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:49:31.064 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:49:31.064 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:49:31.064 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:49:31.065 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:49:31.065 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:49:31.065 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:49:31.065 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:49:31.065 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:49:31.065 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:49:31.065 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:49:31.065 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:49:31.065 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:49:31.065 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:49:31.066 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:49:31.066 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:49:31.066 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:49:31.066 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:49:31.066 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:49:31.066 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:49:31.066 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:49:31.066 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:49:31.066 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:49:31.118 INFO:teuthology.orchestra.run.smithi118.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-29T10:49:32.027 INFO:teuthology.orchestra.run.smithi118.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 221 kB/s | 195 kB 00:00 2024-03-29T10:49:32.144 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:49:32.144 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:32.145 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-03-29T10:49:32.145 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:32.145 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:32.145 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:32.145 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:49:32.145 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:32.145 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:32.145 INFO:teuthology.orchestra.run.smithi112.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:49:32.145 INFO:teuthology.orchestra.run.smithi112.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:49:32.145 INFO:teuthology.orchestra.run.smithi112.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:49:32.145 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:49:32.145 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:49:32.145 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:49:32.146 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:49:32.146 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:49:32.146 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:49:32.146 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:49:32.146 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:49:32.146 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:49:32.146 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:49:32.146 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:49:32.146 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:49:32.146 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:49:32.146 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:49:32.146 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:49:32.146 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:49:32.147 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:49:32.147 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:49:32.147 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:49:32.147 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:49:32.147 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:49:32.147 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:49:32.147 INFO:teuthology.orchestra.run.smithi112.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:49:32.147 INFO:teuthology.orchestra.run.smithi112.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:49:32.147 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:49:32.147 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:49:32.147 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:49:32.147 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:32.147 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:49:32.510 INFO:teuthology.orchestra.run.smithi118.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 453 kB/s | 803 kB 00:01 2024-03-29T10:49:32.562 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-base 2024-03-29T10:49:32.952 INFO:teuthology.orchestra.run.smithi118.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 265 kB/s | 245 kB 00:00 2024-03-29T10:49:33.132 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:13 ago on Fri 29 Mar 2024 10:48:20 AM UTC. 2024-03-29T10:49:33.231 INFO:teuthology.orchestra.run.smithi112.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:49:33.285 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:49:33.286 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-03-29T10:49:33.286 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:49:33.311 INFO:teuthology.orchestra.run.smithi118.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 2.3 MB/s | 7.4 MB 00:03 2024-03-29T10:49:33.314 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:33.314 INFO:teuthology.orchestra.run.smithi118.stdout:Total 11 MB/s | 37 MB 00:03 2024-03-29T10:49:33.350 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install cephadm 2024-03-29T10:49:33.356 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-29T10:49:33.380 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-29T10:49:33.381 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-29T10:49:33.773 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-29T10:49:33.773 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-29T10:49:33.910 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:13 ago on Fri 29 Mar 2024 10:48:20 AM UTC. 2024-03-29T10:49:34.062 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:49:34.062 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:49:34.062 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-03-29T10:49:34.063 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:49:34.063 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-03-29T10:49:34.063 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:49:34.063 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:34.063 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-03-29T10:49:34.063 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:49:34.063 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-03-29T10:49:34.063 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:34.063 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 747 k 2024-03-29T10:49:34.063 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 754 k 2024-03-29T10:49:34.063 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-03-29T10:49:34.473 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-29T10:49:34.559 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:49:34.590 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:49:34.653 INFO:teuthology.orchestra.run.smithi112.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.2 MB/s | 747 kB 00:00 2024-03-29T10:49:34.654 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:34.654 INFO:teuthology.orchestra.run.smithi112.stdout:Total 1.2 MB/s | 747 kB 00:00 2024-03-29T10:49:34.654 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-03-29T10:49:34.660 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-03-29T10:49:34.660 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-03-29T10:49:34.666 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-03-29T10:49:34.666 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-03-29T10:49:34.685 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:49:34.739 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-03-29T10:49:34.745 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:49:34.923 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:49:34.970 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:49:35.051 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:49:35.221 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:49:35.319 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:49:35.699 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:49:35.715 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:49:35.814 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:49:35.998 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:49:35.999 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:35.999 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-03-29T10:49:35.999 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:49:35.999 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:35.999 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:49:36.208 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:49:36.775 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:16 ago on Fri 29 Mar 2024 10:48:20 AM UTC. 2024-03-29T10:49:36.928 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:49:36.928 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:49:36.929 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-03-29T10:49:36.929 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:49:36.929 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-03-29T10:49:36.929 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:49:36.929 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:36.929 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-03-29T10:49:36.929 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:49:36.929 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-03-29T10:49:36.929 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:36.929 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 145 k 2024-03-29T10:49:36.930 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 436 k 2024-03-29T10:49:36.930 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-03-29T10:49:37.045 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:49:37.467 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 270 kB/s | 145 kB 00:00 2024-03-29T10:49:37.468 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:37.468 INFO:teuthology.orchestra.run.smithi112.stdout:Total 270 kB/s | 145 kB 00:00 2024-03-29T10:49:37.468 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-03-29T10:49:37.475 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-03-29T10:49:37.476 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-03-29T10:49:37.528 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-03-29T10:49:37.529 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-03-29T10:49:37.676 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-03-29T10:49:37.743 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:49:37.767 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:49:37.767 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:49:37.768 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:49:37.768 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:38.121 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:49:38.161 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:49:38.747 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:49:38.747 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:38.747 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-03-29T10:49:38.747 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:38.747 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:38.747 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:49:38.987 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr 2024-03-29T10:49:39.562 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:19 ago on Fri 29 Mar 2024 10:48:20 AM UTC. 2024-03-29T10:49:39.664 INFO:teuthology.orchestra.run.smithi112.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:49:39.716 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:49:39.717 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-03-29T10:49:39.717 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:49:39.781 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:49:40.339 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:20 ago on Fri 29 Mar 2024 10:48:20 AM UTC. 2024-03-29T10:49:40.490 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:49:40.494 INFO:teuthology.orchestra.run.smithi112.stdout:====================================================================================== 2024-03-29T10:49:40.494 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-03-29T10:49:40.494 INFO:teuthology.orchestra.run.smithi112.stdout:====================================================================================== 2024-03-29T10:49:40.494 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-03-29T10:49:40.494 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:49:40.494 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-03-29T10:49:40.494 INFO:teuthology.orchestra.run.smithi112.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:49:40.494 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:49:40.494 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:49:40.494 INFO:teuthology.orchestra.run.smithi112.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:49:40.495 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:49:40.495 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:49:40.495 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:49:40.495 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:49:40.495 INFO:teuthology.orchestra.run.smithi112.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:49:40.495 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:49:40.495 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:49:40.495 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:40.495 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-03-29T10:49:40.495 INFO:teuthology.orchestra.run.smithi112.stdout:====================================================================================== 2024-03-29T10:49:40.495 INFO:teuthology.orchestra.run.smithi112.stdout:Install 12 Packages 2024-03-29T10:49:40.495 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:40.495 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 6.9 M 2024-03-29T10:49:40.495 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 26 M 2024-03-29T10:49:40.495 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-03-29T10:49:40.664 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:49:40.665 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:49:40.665 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:49:40.665 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:49:40.665 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:49:40.665 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:49:40.665 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:49:40.665 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:49:40.665 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:49:40.665 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:49:40.665 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:49:40.665 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:49:41.094 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:49:41.095 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:41.095 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-29T10:49:41.095 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:49:41.095 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:49:41.095 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:49:41.095 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:49:41.095 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:49:41.095 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:49:41.095 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:49:41.095 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:49:41.096 INFO:teuthology.orchestra.run.smithi118.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:49:41.096 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:49:41.096 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:49:41.096 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:49:41.096 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:41.096 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:49:41.105 INFO:teuthology.orchestra.run.smithi112.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 33 kB/s | 15 kB 00:00 2024-03-29T10:49:41.121 INFO:teuthology.orchestra.run.smithi112.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 65 kB/s | 31 kB 00:00 2024-03-29T10:49:41.371 INFO:teuthology.orchestra.run.smithi112.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 952 kB/s | 238 kB 00:00 2024-03-29T10:49:41.486 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:49:41.505 INFO:teuthology.orchestra.run.smithi112.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.5 MB/s | 1.0 MB 00:00 2024-03-29T10:49:41.922 INFO:teuthology.orchestra.run.smithi112.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.3 MB/s | 548 kB 00:00 2024-03-29T10:49:41.964 INFO:teuthology.orchestra.run.smithi112.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 466 kB/s | 19 kB 00:00 2024-03-29T10:49:42.039 INFO:teuthology.orchestra.run.smithi112.stdout:(7/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 1.3 MB/s | 1.8 MB 00:01 2024-03-29T10:49:42.064 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:23 ago on Fri 29 Mar 2024 10:48:19 AM UTC. 2024-03-29T10:49:42.148 INFO:teuthology.orchestra.run.smithi112.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 11 MB/s | 2.0 MB 00:00 2024-03-29T10:49:42.218 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:49:42.219 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2024-03-29T10:49:42.219 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-29T10:49:42.220 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2024-03-29T10:49:42.220 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-29T10:49:42.220 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:49:42.220 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-29T10:49:42.220 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:49:42.220 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:49:42.220 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:49:42.220 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:49:42.220 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:49:42.220 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:49:42.220 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:49:42.220 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:49:42.221 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:49:42.221 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:49:42.221 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:49:42.221 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:49:42.221 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:42.221 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-29T10:49:42.221 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2024-03-29T10:49:42.221 INFO:teuthology.orchestra.run.smithi118.stdout:Install 13 Packages 2024-03-29T10:49:42.221 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:42.222 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 2.0 M 2024-03-29T10:49:42.222 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 26 M 2024-03-29T10:49:42.222 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-29T10:49:42.231 INFO:teuthology.orchestra.run.smithi112.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 753 kB/s | 144 kB 00:00 2024-03-29T10:49:42.323 INFO:teuthology.orchestra.run.smithi112.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-03-29T10:49:42.386 INFO:teuthology.orchestra.run.smithi118.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 162 kB/s | 18 kB 00:00 2024-03-29T10:49:42.436 INFO:teuthology.orchestra.run.smithi118.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 151 kB/s | 24 kB 00:00 2024-03-29T10:49:42.479 INFO:teuthology.orchestra.run.smithi118.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-29T10:49:42.504 INFO:teuthology.orchestra.run.smithi118.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-29T10:49:42.529 INFO:teuthology.orchestra.run.smithi118.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T10:49:42.554 INFO:teuthology.orchestra.run.smithi118.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 869 kB/s | 43 kB 00:00 2024-03-29T10:49:42.738 INFO:teuthology.orchestra.run.smithi118.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 154 kB/s | 32 kB 00:00 2024-03-29T10:49:42.763 INFO:teuthology.orchestra.run.smithi118.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 72 kB/s | 15 kB 00:00 2024-03-29T10:49:42.822 INFO:teuthology.orchestra.run.smithi118.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 95 kB/s | 52 kB 00:00 2024-03-29T10:49:42.941 INFO:teuthology.orchestra.run.smithi118.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.1 MB/s | 203 kB 00:00 2024-03-29T10:49:43.022 INFO:teuthology.orchestra.run.smithi118.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.1 MB/s | 90 kB 00:00 2024-03-29T10:49:43.048 INFO:teuthology.orchestra.run.smithi118.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 262 kB/s | 59 kB 00:00 2024-03-29T10:49:43.115 INFO:teuthology.orchestra.run.smithi118.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.9 MB/s | 1.0 MB 00:00 2024-03-29T10:49:43.116 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:43.117 INFO:teuthology.orchestra.run.smithi118.stdout:Total 2.3 MB/s | 2.0 MB 00:00 2024-03-29T10:49:43.117 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-29T10:49:43.130 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-29T10:49:43.130 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-29T10:49:43.140 INFO:teuthology.orchestra.run.smithi112.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 488 kB/s | 863 kB 00:01 2024-03-29T10:49:43.165 INFO:teuthology.orchestra.run.smithi112.stdout:(12/12): python3-repoze-lru-0.7-16.el9.noarch.r 30 kB/s | 31 kB 00:01 2024-03-29T10:49:43.169 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:43.169 INFO:teuthology.orchestra.run.smithi112.stdout:Total 2.6 MB/s | 6.9 MB 00:02 2024-03-29T10:49:43.190 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-03-29T10:49:43.209 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-03-29T10:49:43.209 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-03-29T10:49:43.222 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-29T10:49:43.223 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-29T10:49:43.392 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-03-29T10:49:43.392 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-03-29T10:49:43.743 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-29T10:49:43.893 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-03-29T10:49:43.912 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:49:43.982 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:49:43.983 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:49:44.017 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:49:44.037 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:49:44.064 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:49:44.075 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:49:44.094 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:49:44.117 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:49:44.188 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:49:44.210 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:49:44.311 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:49:44.328 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:49:44.377 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:49:44.405 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:49:44.448 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:49:44.478 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:49:44.501 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:49:44.741 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:49:44.767 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:49:45.093 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:49:45.158 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:49:45.201 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:49:45.248 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:49:45.270 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:49:45.630 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:49:45.631 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:49:45.631 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:49:45.631 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:49:45.631 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:49:45.631 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:49:45.631 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:49:45.631 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:49:45.631 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:49:45.631 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:49:45.631 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:49:45.631 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:49:45.978 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:49:45.978 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:45.979 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-03-29T10:49:45.979 INFO:teuthology.orchestra.run.smithi112.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:49:45.979 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:49:45.979 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:49:45.979 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:49:45.979 INFO:teuthology.orchestra.run.smithi112.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:49:45.979 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:49:45.979 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:49:45.979 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:49:45.979 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:49:45.979 INFO:teuthology.orchestra.run.smithi112.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:49:45.979 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:49:45.979 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:49:45.979 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:45.979 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:49:46.054 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:49:46.054 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:49:46.054 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:49:46.055 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:49:46.055 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:49:46.055 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:49:46.055 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:49:46.055 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:49:46.055 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:49:46.055 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:49:46.055 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:49:46.055 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:49:46.055 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:49:46.216 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:49:46.417 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:49:46.418 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:46.418 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-29T10:49:46.418 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:49:46.418 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:49:46.418 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:49:46.418 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:49:46.418 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:49:46.418 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:49:46.418 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:49:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:49:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:49:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:49:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:49:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:49:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:49:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:46.419 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:49:46.663 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:49:46.797 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:26 ago on Fri 29 Mar 2024 10:48:20 AM UTC. 2024-03-29T10:49:46.961 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:49:46.962 INFO:teuthology.orchestra.run.smithi112.stdout:============================================================================================ 2024-03-29T10:49:46.962 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-03-29T10:49:46.962 INFO:teuthology.orchestra.run.smithi112.stdout:============================================================================================ 2024-03-29T10:49:46.962 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-03-29T10:49:46.962 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:49:46.962 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-03-29T10:49:46.962 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:49:46.962 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:49:46.962 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:49:46.963 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:49:46.963 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:49:46.963 INFO:teuthology.orchestra.run.smithi112.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:49:46.963 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:49:46.963 INFO:teuthology.orchestra.run.smithi112.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:49:46.963 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:49:46.963 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:49:46.963 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:49:46.963 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:46.963 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-03-29T10:49:46.964 INFO:teuthology.orchestra.run.smithi112.stdout:============================================================================================ 2024-03-29T10:49:46.964 INFO:teuthology.orchestra.run.smithi112.stdout:Install 12 Packages 2024-03-29T10:49:46.964 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:46.965 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 37 M 2024-03-29T10:49:46.965 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 209 M 2024-03-29T10:49:46.965 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-03-29T10:49:47.231 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:28 ago on Fri 29 Mar 2024 10:48:19 AM UTC. 2024-03-29T10:49:47.384 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:49:47.385 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2024-03-29T10:49:47.385 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-29T10:49:47.385 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2024-03-29T10:49:47.385 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-29T10:49:47.385 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:49:47.385 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-29T10:49:47.385 INFO:teuthology.orchestra.run.smithi118.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:49:47.385 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:49:47.385 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:49:47.386 INFO:teuthology.orchestra.run.smithi118.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:49:47.386 INFO:teuthology.orchestra.run.smithi118.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:49:47.386 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:47.386 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-29T10:49:47.386 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2024-03-29T10:49:47.386 INFO:teuthology.orchestra.run.smithi118.stdout:Install 6 Packages 2024-03-29T10:49:47.386 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:47.387 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 6.8 M 2024-03-29T10:49:47.387 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 33 M 2024-03-29T10:49:47.387 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-29T10:49:47.497 INFO:teuthology.orchestra.run.smithi112.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 212 kB/s | 32 kB 00:00 2024-03-29T10:49:47.530 INFO:teuthology.orchestra.run.smithi112.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 522 kB/s | 17 kB 00:00 2024-03-29T10:49:47.564 INFO:teuthology.orchestra.run.smithi112.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-29T10:49:47.731 INFO:teuthology.orchestra.run.smithi112.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2024-03-29T10:49:47.848 INFO:teuthology.orchestra.run.smithi112.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 41 MB/s | 4.8 MB 00:00 2024-03-29T10:49:47.857 INFO:teuthology.orchestra.run.smithi118.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-29T10:49:47.907 INFO:teuthology.orchestra.run.smithi112.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.3 MB/s | 3.0 MB 00:00 2024-03-29T10:49:47.932 INFO:teuthology.orchestra.run.smithi112.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-29T10:49:48.119 INFO:teuthology.orchestra.run.smithi118.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.0 MB/s | 548 kB 00:00 2024-03-29T10:49:48.161 INFO:teuthology.orchestra.run.smithi118.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 391 kB/s | 248 kB 00:00 2024-03-29T10:49:48.203 INFO:teuthology.orchestra.run.smithi118.stdout:(4/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 204 kB/s | 138 kB 00:00 2024-03-29T10:49:48.224 INFO:teuthology.orchestra.run.smithi112.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 49 MB/s | 16 MB 00:00 2024-03-29T10:49:48.236 INFO:teuthology.orchestra.run.smithi118.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 492 kB/s | 58 kB 00:00 2024-03-29T10:49:48.278 INFO:teuthology.orchestra.run.smithi118.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 436 kB/s | 51 kB 00:00 2024-03-29T10:49:48.280 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:48.280 INFO:teuthology.orchestra.run.smithi118.stdout:Total 7.6 MB/s | 6.8 MB 00:00 2024-03-29T10:49:48.294 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-29T10:49:48.304 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-29T10:49:48.304 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-29T10:49:48.393 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-29T10:49:48.394 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-29T10:49:48.449 INFO:teuthology.orchestra.run.smithi112.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-03-29T10:49:48.475 INFO:teuthology.orchestra.run.smithi112.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 781 kB/s | 195 kB 00:00 2024-03-29T10:49:48.599 INFO:teuthology.orchestra.run.smithi112.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-03-29T10:49:48.691 INFO:teuthology.orchestra.run.smithi112.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 5.5 MB/s | 7.4 MB 00:01 2024-03-29T10:49:48.694 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:48.694 INFO:teuthology.orchestra.run.smithi112.stdout:Total 21 MB/s | 37 MB 00:01 2024-03-29T10:49:48.696 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-29T10:49:48.735 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-03-29T10:49:48.759 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-03-29T10:49:48.759 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-03-29T10:49:48.876 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:49:48.921 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:49:49.119 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-03-29T10:49:49.119 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-03-29T10:49:49.433 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:49:49.533 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:49:49.596 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:49:49.619 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:49:49.823 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-03-29T10:49:49.897 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:49:49.929 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:49:50.021 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:49:50.065 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:49:50.241 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:49:50.264 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:49:50.348 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:49:50.409 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:49:50.409 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:49:50.409 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:49:50.410 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:49:50.410 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:49:50.410 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:49:50.723 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:49:50.723 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:50.723 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-29T10:49:50.723 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:49:50.723 INFO:teuthology.orchestra.run.smithi118.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:49:50.723 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:49:50.723 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:49:50.723 INFO:teuthology.orchestra.run.smithi118.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:49:50.723 INFO:teuthology.orchestra.run.smithi118.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:49:50.723 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:50.723 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:49:50.944 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-fuse 2024-03-29T10:49:50.988 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:49:51.085 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:49:51.504 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 10:48:19 AM UTC. 2024-03-29T10:49:51.653 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:49:51.654 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:49:51.654 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-29T10:49:51.654 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:49:51.654 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-29T10:49:51.654 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:49:51.654 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-29T10:49:51.654 INFO:teuthology.orchestra.run.smithi118.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:49:51.654 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:51.654 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-29T10:49:51.655 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:49:51.655 INFO:teuthology.orchestra.run.smithi118.stdout:Install 2 Packages 2024-03-29T10:49:51.655 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:51.655 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 964 k 2024-03-29T10:49:51.655 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 2.8 M 2024-03-29T10:49:51.656 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-29T10:49:51.799 INFO:teuthology.orchestra.run.smithi118.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 546 kB/s | 79 kB 00:00 2024-03-29T10:49:52.314 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:49:52.541 INFO:teuthology.orchestra.run.smithi118.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.0 MB/s | 885 kB 00:00 2024-03-29T10:49:52.542 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:52.543 INFO:teuthology.orchestra.run.smithi118.stdout:Total 1.1 MB/s | 964 kB 00:00 2024-03-29T10:49:52.543 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-29T10:49:52.563 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-29T10:49:52.563 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-29T10:49:52.636 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-29T10:49:52.636 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-29T10:49:52.819 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-29T10:49:52.950 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:49:53.033 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:49:53.346 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:49:53.374 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:49:53.749 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:49:53.749 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:49:54.039 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:49:54.039 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:54.039 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-29T10:49:54.039 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:49:54.039 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:54.040 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:49:54.335 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-volume 2024-03-29T10:49:54.909 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 10:48:19 AM UTC. 2024-03-29T10:49:55.062 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:49:55.063 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:49:55.063 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-29T10:49:55.063 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:49:55.063 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-29T10:49:55.063 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:49:55.063 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-29T10:49:55.063 INFO:teuthology.orchestra.run.smithi118.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:49:55.063 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:55.063 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-29T10:49:55.063 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:49:55.063 INFO:teuthology.orchestra.run.smithi118.stdout:Install 2 Packages 2024-03-29T10:49:55.064 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:55.064 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 332 k 2024-03-29T10:49:55.064 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 1.5 M 2024-03-29T10:49:55.064 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-29T10:49:55.196 INFO:teuthology.orchestra.run.smithi118.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 522 kB/s | 69 kB 00:00 2024-03-29T10:49:55.801 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:49:55.801 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:49:55.802 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:49:55.802 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:49:55.802 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:49:55.802 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:49:55.802 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:49:55.802 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:49:55.802 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:49:55.802 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:49:55.802 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:49:55.802 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:49:55.896 INFO:teuthology.orchestra.run.smithi118.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 317 kB/s | 264 kB 00:00 2024-03-29T10:49:55.897 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:55.898 INFO:teuthology.orchestra.run.smithi118.stdout:Total 398 kB/s | 332 kB 00:00 2024-03-29T10:49:55.898 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-29T10:49:55.907 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-29T10:49:55.907 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-29T10:49:55.943 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-29T10:49:55.944 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-29T10:49:56.090 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-29T10:49:56.251 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:49:56.251 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:56.252 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-03-29T10:49:56.252 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:49:56.252 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:49:56.252 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:49:56.252 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:49:56.252 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:49:56.252 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:49:56.252 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:49:56.252 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:49:56.252 INFO:teuthology.orchestra.run.smithi112.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:49:56.252 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:49:56.252 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:49:56.252 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:49:56.252 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:56.253 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:49:56.253 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:49:56.308 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:49:56.324 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:49:56.324 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:49:56.324 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:49:56.324 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:56.618 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:49:57.061 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:49:57.192 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 10:48:20 AM UTC. 2024-03-29T10:49:57.347 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:49:57.348 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================== 2024-03-29T10:49:57.348 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-03-29T10:49:57.348 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================== 2024-03-29T10:49:57.348 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-03-29T10:49:57.348 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:49:57.348 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-03-29T10:49:57.348 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:49:57.349 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:49:57.349 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:49:57.349 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:49:57.349 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:49:57.349 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:49:57.349 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:49:57.349 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:49:57.349 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:49:57.349 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:49:57.349 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:49:57.349 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:49:57.349 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:57.349 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-03-29T10:49:57.350 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================== 2024-03-29T10:49:57.350 INFO:teuthology.orchestra.run.smithi112.stdout:Install 13 Packages 2024-03-29T10:49:57.350 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:49:57.350 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 2.0 M 2024-03-29T10:49:57.350 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 26 M 2024-03-29T10:49:57.351 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-03-29T10:49:57.389 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:49:57.389 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:57.389 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-29T10:49:57.389 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:49:57.389 INFO:teuthology.orchestra.run.smithi118.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:49:57.389 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:57.389 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:49:57.654 INFO:teuthology.orchestra.run.smithi112.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 207 kB/s | 24 kB 00:00 2024-03-29T10:49:57.679 INFO:teuthology.orchestra.run.smithi112.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 126 kB/s | 18 kB 00:00 2024-03-29T10:49:57.719 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librados-devel 2024-03-29T10:49:57.729 INFO:teuthology.orchestra.run.smithi112.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T10:49:57.755 INFO:teuthology.orchestra.run.smithi112.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T10:49:57.780 INFO:teuthology.orchestra.run.smithi112.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T10:49:57.805 INFO:teuthology.orchestra.run.smithi112.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 860 kB/s | 43 kB 00:00 2024-03-29T10:49:57.980 INFO:teuthology.orchestra.run.smithi112.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 118 kB/s | 52 kB 00:00 2024-03-29T10:49:58.081 INFO:teuthology.orchestra.run.smithi112.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 107 kB/s | 32 kB 00:00 2024-03-29T10:49:58.106 INFO:teuthology.orchestra.run.smithi112.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 50 kB/s | 15 kB 00:00 2024-03-29T10:49:58.299 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 10:48:19 AM UTC. 2024-03-29T10:49:58.450 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:49:58.450 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:49:58.450 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2024-03-29T10:49:58.450 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:49:58.450 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-29T10:49:58.450 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:49:58.450 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:58.450 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-29T10:49:58.450 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:49:58.450 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-03-29T10:49:58.450 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:49:58.451 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 127 k 2024-03-29T10:49:58.451 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 456 k 2024-03-29T10:49:58.451 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-29T10:49:58.456 INFO:teuthology.orchestra.run.smithi112.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 169 kB/s | 59 kB 00:00 2024-03-29T10:49:58.507 INFO:teuthology.orchestra.run.smithi112.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 385 kB/s | 203 kB 00:00 2024-03-29T10:49:58.782 INFO:teuthology.orchestra.run.smithi112.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 1.5 MB/s | 1.0 MB 00:00 2024-03-29T10:49:58.807 INFO:teuthology.orchestra.run.smithi112.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 256 kB/s | 90 kB 00:00 2024-03-29T10:49:58.809 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:58.810 INFO:teuthology.orchestra.run.smithi112.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2024-03-29T10:49:58.810 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-03-29T10:49:58.823 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-03-29T10:49:58.824 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-03-29T10:49:58.904 INFO:teuthology.orchestra.run.smithi118.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 280 kB/s | 127 kB 00:00 2024-03-29T10:49:58.904 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:58.904 INFO:teuthology.orchestra.run.smithi118.stdout:Total 279 kB/s | 127 kB 00:00 2024-03-29T10:49:58.905 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-29T10:49:58.910 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-29T10:49:58.910 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-29T10:49:58.915 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-03-29T10:49:58.916 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-03-29T10:49:58.967 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-29T10:49:58.967 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-29T10:49:59.125 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-29T10:49:59.236 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:49:59.426 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-03-29T10:49:59.602 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:49:59.657 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:49:59.712 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:49:59.728 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:49:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:49:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:49:59.876 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:49:59.976 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:50:00.018 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:00.018 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:00.018 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-29T10:50:00.019 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:00.019 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:00.019 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:50:00.026 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:50:00.297 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install libcephfs2 2024-03-29T10:50:00.744 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:50:00.789 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:50:00.823 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:50:00.872 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:50:00.946 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:48:19 AM UTC. 2024-03-29T10:50:00.947 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:50:00.999 INFO:teuthology.orchestra.run.smithi118.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:01.056 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:50:01.056 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-29T10:50:01.057 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:50:01.120 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install libcephfs-devel 2024-03-29T10:50:01.686 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:48:19 AM UTC. 2024-03-29T10:50:01.697 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:50:01.697 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:50:01.697 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:50:01.697 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:50:01.697 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:50:01.698 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:50:01.698 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:50:01.698 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:50:01.698 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:50:01.698 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:50:01.698 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:50:01.698 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:50:01.698 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:50:01.836 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:50:01.837 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:50:01.837 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2024-03-29T10:50:01.837 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:50:01.837 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-29T10:50:01.837 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:50:01.837 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:01.837 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-29T10:50:01.837 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:50:01.837 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-03-29T10:50:01.837 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:01.838 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 31 k 2024-03-29T10:50:01.838 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 140 k 2024-03-29T10:50:01.838 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-29T10:50:02.044 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:50:02.044 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:02.044 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-03-29T10:50:02.044 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:02.044 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:50:02.044 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:50:02.044 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:50:02.044 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:50:02.045 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:50:02.045 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:50:02.045 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:50:02.045 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:50:02.045 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:50:02.045 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:50:02.045 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:50:02.045 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:50:02.045 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:02.045 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:50:02.176 INFO:teuthology.orchestra.run.smithi118.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 92 kB/s | 31 kB 00:00 2024-03-29T10:50:02.176 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:02.177 INFO:teuthology.orchestra.run.smithi118.stdout:Total 92 kB/s | 31 kB 00:00 2024-03-29T10:50:02.177 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-29T10:50:02.180 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-29T10:50:02.180 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-29T10:50:02.211 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-29T10:50:02.211 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-29T10:50:02.303 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:50:02.320 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-29T10:50:02.433 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:02.870 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:48:20 AM UTC. 2024-03-29T10:50:02.895 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:03.020 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:50:03.021 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================== 2024-03-29T10:50:03.021 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-03-29T10:50:03.021 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================== 2024-03-29T10:50:03.021 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-03-29T10:50:03.021 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:50:03.022 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-03-29T10:50:03.022 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:50:03.022 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:50:03.022 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:50:03.022 INFO:teuthology.orchestra.run.smithi112.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:50:03.022 INFO:teuthology.orchestra.run.smithi112.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:50:03.022 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:03.022 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-03-29T10:50:03.022 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================== 2024-03-29T10:50:03.022 INFO:teuthology.orchestra.run.smithi112.stdout:Install 6 Packages 2024-03-29T10:50:03.022 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:03.023 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 6.8 M 2024-03-29T10:50:03.023 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 33 M 2024-03-29T10:50:03.023 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-03-29T10:50:03.161 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:03.162 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:03.162 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-29T10:50:03.162 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:03.162 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:03.162 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:50:03.344 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librados2 2024-03-29T10:50:03.483 INFO:teuthology.orchestra.run.smithi112.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 5.8 MB 00:00 2024-03-29T10:50:03.508 INFO:teuthology.orchestra.run.smithi112.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 495 kB/s | 138 kB 00:00 2024-03-29T10:50:03.583 INFO:teuthology.orchestra.run.smithi112.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 701 kB/s | 248 kB 00:00 2024-03-29T10:50:03.667 INFO:teuthology.orchestra.run.smithi112.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 364 kB/s | 58 kB 00:00 2024-03-29T10:50:03.734 INFO:teuthology.orchestra.run.smithi112.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 340 kB/s | 51 kB 00:00 2024-03-29T10:50:03.776 INFO:teuthology.orchestra.run.smithi112.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2024-03-29T10:50:03.776 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:03.777 INFO:teuthology.orchestra.run.smithi112.stdout:Total 9.0 MB/s | 6.8 MB 00:00 2024-03-29T10:50:03.790 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-03-29T10:50:03.800 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-03-29T10:50:03.800 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-03-29T10:50:03.887 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-03-29T10:50:03.888 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-03-29T10:50:03.910 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:48:19 AM UTC. 2024-03-29T10:50:04.007 INFO:teuthology.orchestra.run.smithi118.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:04.059 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:50:04.059 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-29T10:50:04.060 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:50:04.132 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librbd1 2024-03-29T10:50:04.191 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-03-29T10:50:04.348 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:50:04.401 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:50:04.693 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 10:48:19 AM UTC. 2024-03-29T10:50:04.790 INFO:teuthology.orchestra.run.smithi118.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:04.840 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:50:04.841 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-29T10:50:04.842 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:50:04.892 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rados 2024-03-29T10:50:04.909 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:50:04.988 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:50:05.057 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:50:05.081 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:50:05.454 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 10:48:19 AM UTC. 2024-03-29T10:50:05.552 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:05.603 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:50:05.604 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-29T10:50:05.604 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:50:05.666 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rgw 2024-03-29T10:50:05.869 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:50:05.869 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:50:05.869 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:50:05.869 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:50:05.869 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:50:05.869 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:50:06.161 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:50:06.161 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:06.161 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-03-29T10:50:06.161 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:06.161 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:50:06.162 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:50:06.162 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:50:06.162 INFO:teuthology.orchestra.run.smithi112.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:50:06.162 INFO:teuthology.orchestra.run.smithi112.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:50:06.162 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:06.162 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:50:06.232 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 10:48:19 AM UTC. 2024-03-29T10:50:06.330 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:06.382 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:50:06.383 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-29T10:50:06.383 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:50:06.387 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-fuse 2024-03-29T10:50:06.458 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-cephfs 2024-03-29T10:50:06.944 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 10:48:20 AM UTC. 2024-03-29T10:50:07.026 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 10:48:19 AM UTC. 2024-03-29T10:50:07.095 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:50:07.096 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:50:07.096 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-03-29T10:50:07.096 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:50:07.096 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-03-29T10:50:07.096 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:50:07.096 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-03-29T10:50:07.096 INFO:teuthology.orchestra.run.smithi112.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:50:07.097 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:07.097 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-03-29T10:50:07.097 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:50:07.097 INFO:teuthology.orchestra.run.smithi112.stdout:Install 2 Packages 2024-03-29T10:50:07.097 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:07.097 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 964 k 2024-03-29T10:50:07.097 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 2.8 M 2024-03-29T10:50:07.097 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-03-29T10:50:07.124 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:07.175 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:50:07.176 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-29T10:50:07.176 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:50:07.218 INFO:teuthology.orchestra.run.smithi112.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 647 kB/s | 79 kB 00:00 2024-03-29T10:50:07.251 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rbd 2024-03-29T10:50:07.410 INFO:teuthology.orchestra.run.smithi112.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.8 MB/s | 885 kB 00:00 2024-03-29T10:50:07.412 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:07.412 INFO:teuthology.orchestra.run.smithi112.stdout:Total 3.0 MB/s | 964 kB 00:00 2024-03-29T10:50:07.412 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-03-29T10:50:07.432 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-03-29T10:50:07.432 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-03-29T10:50:07.505 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-03-29T10:50:07.505 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-03-29T10:50:07.690 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-03-29T10:50:07.829 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:50:07.872 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 10:48:19 AM UTC. 2024-03-29T10:50:07.901 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:50:07.970 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:08.021 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:50:08.022 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-29T10:50:08.022 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:50:08.091 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-fuse 2024-03-29T10:50:08.586 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:50:08.587 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:50:08.646 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 10:48:19 AM UTC. 2024-03-29T10:50:08.792 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:50:08.792 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:50:08.793 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2024-03-29T10:50:08.793 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:50:08.793 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-29T10:50:08.793 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:50:08.793 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:08.793 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-29T10:50:08.793 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:50:08.793 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-03-29T10:50:08.793 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:08.794 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 85 k 2024-03-29T10:50:08.794 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 231 k 2024-03-29T10:50:08.794 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-29T10:50:08.935 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:50:08.936 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:08.936 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-03-29T10:50:08.936 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:50:08.936 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:08.936 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:50:09.079 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 299 kB/s | 85 kB 00:00 2024-03-29T10:50:09.079 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:09.080 INFO:teuthology.orchestra.run.smithi118.stdout:Total 298 kB/s | 85 kB 00:00 2024-03-29T10:50:09.080 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-29T10:50:09.086 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-29T10:50:09.087 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-29T10:50:09.142 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-29T10:50:09.142 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-29T10:50:09.225 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-volume 2024-03-29T10:50:09.279 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-29T10:50:09.410 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:09.801 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:09.804 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 10:48:20 AM UTC. 2024-03-29T10:50:09.958 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:50:09.959 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:50:09.959 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-03-29T10:50:09.959 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:50:09.959 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-03-29T10:50:09.959 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:50:09.959 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-03-29T10:50:09.959 INFO:teuthology.orchestra.run.smithi112.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:50:09.959 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:09.959 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-03-29T10:50:09.959 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:50:09.959 INFO:teuthology.orchestra.run.smithi112.stdout:Install 2 Packages 2024-03-29T10:50:09.959 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:09.960 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 332 k 2024-03-29T10:50:09.960 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 1.5 M 2024-03-29T10:50:09.960 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-03-29T10:50:10.081 INFO:teuthology.orchestra.run.smithi112.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 569 kB/s | 69 kB 00:00 2024-03-29T10:50:10.094 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:10.094 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:10.094 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-29T10:50:10.095 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:10.095 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:10.095 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:50:10.344 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-mirror 2024-03-29T10:50:10.398 INFO:teuthology.orchestra.run.smithi112.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 603 kB/s | 264 kB 00:00 2024-03-29T10:50:10.399 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:10.400 INFO:teuthology.orchestra.run.smithi112.stdout:Total 755 kB/s | 332 kB 00:00 2024-03-29T10:50:10.400 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-03-29T10:50:10.408 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-03-29T10:50:10.408 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-03-29T10:50:10.445 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-03-29T10:50:10.446 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-03-29T10:50:10.592 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-03-29T10:50:10.762 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:50:10.800 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:50:10.816 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:50:10.816 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:50:10.816 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:50:10.816 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:10.921 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Mar 2024 10:48:19 AM UTC. 2024-03-29T10:50:11.080 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:50:11.080 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:50:11.080 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2024-03-29T10:50:11.081 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:50:11.081 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-29T10:50:11.081 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:50:11.081 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:11.081 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-29T10:50:11.081 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:50:11.081 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-03-29T10:50:11.081 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:11.081 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 3.1 M 2024-03-29T10:50:11.081 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 13 M 2024-03-29T10:50:11.081 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-29T10:50:11.528 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:50:11.533 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 6.8 MB/s | 3.1 MB 00:00 2024-03-29T10:50:11.533 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:11.534 INFO:teuthology.orchestra.run.smithi118.stdout:Total 6.8 MB/s | 3.1 MB 00:00 2024-03-29T10:50:11.534 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-29T10:50:11.544 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-29T10:50:11.544 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-29T10:50:11.617 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-29T10:50:11.617 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-29T10:50:11.877 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:50:11.877 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:11.877 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-03-29T10:50:11.877 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:11.877 INFO:teuthology.orchestra.run.smithi112.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:50:11.877 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:11.877 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:50:12.050 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-29T10:50:12.100 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:12.125 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:12.126 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:50:12.126 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:50:12.126 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:50:12.126 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:50:12.126 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:12.217 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librados-devel 2024-03-29T10:50:12.794 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:52 ago on Fri 29 Mar 2024 10:48:20 AM UTC. 2024-03-29T10:50:12.944 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:50:12.946 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:50:12.946 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-03-29T10:50:12.946 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:50:12.946 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-03-29T10:50:12.946 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:50:12.946 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:12.947 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-03-29T10:50:12.947 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:50:12.947 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-03-29T10:50:12.947 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:12.947 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 127 k 2024-03-29T10:50:12.947 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 456 k 2024-03-29T10:50:12.948 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-03-29T10:50:13.263 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:13.263 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:13.263 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-29T10:50:13.263 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:13.263 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:13.263 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:50:13.370 INFO:teuthology.orchestra.run.smithi112.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 299 kB/s | 127 kB 00:00 2024-03-29T10:50:13.370 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:13.371 INFO:teuthology.orchestra.run.smithi112.stdout:Total 298 kB/s | 127 kB 00:00 2024-03-29T10:50:13.371 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-03-29T10:50:13.376 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-03-29T10:50:13.376 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-03-29T10:50:13.433 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-03-29T10:50:13.433 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-03-29T10:50:13.522 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-nbd 2024-03-29T10:50:13.580 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-03-29T10:50:13.679 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:14.100 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:55 ago on Fri 29 Mar 2024 10:48:19 AM UTC. 2024-03-29T10:50:14.157 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:14.247 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-29T10:50:14.247 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:50:14.247 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2024-03-29T10:50:14.247 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:50:14.247 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-29T10:50:14.247 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:50:14.248 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:14.248 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-29T10:50:14.248 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-29T10:50:14.248 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-03-29T10:50:14.248 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:14.248 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 171 k 2024-03-29T10:50:14.248 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 494 k 2024-03-29T10:50:14.249 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-29T10:50:14.469 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:14.470 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:14.470 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-03-29T10:50:14.470 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:14.470 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:14.470 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:50:14.637 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 442 kB/s | 171 kB 00:00 2024-03-29T10:50:14.638 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:14.639 INFO:teuthology.orchestra.run.smithi118.stdout:Total 439 kB/s | 171 kB 00:00 2024-03-29T10:50:14.639 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-29T10:50:14.646 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-29T10:50:14.646 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-29T10:50:14.673 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install libcephfs2 2024-03-29T10:50:14.701 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-29T10:50:14.702 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-29T10:50:14.870 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-29T10:50:15.002 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:15.253 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:55 ago on Fri 29 Mar 2024 10:48:20 AM UTC. 2024-03-29T10:50:15.355 INFO:teuthology.orchestra.run.smithi112.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:15.409 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:50:15.410 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-03-29T10:50:15.410 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:50:15.443 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:15.489 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install libcephfs-devel 2024-03-29T10:50:15.737 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:15.737 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:15.737 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-29T10:50:15.738 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:15.738 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:15.738 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-29T10:50:16.001 DEBUG:teuthology.parallel:result is None 2024-03-29T10:50:16.053 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:56 ago on Fri 29 Mar 2024 10:48:20 AM UTC. 2024-03-29T10:50:16.204 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:50:16.205 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:50:16.205 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-03-29T10:50:16.205 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:50:16.205 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-03-29T10:50:16.205 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:50:16.205 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:16.205 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-03-29T10:50:16.205 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:50:16.205 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-03-29T10:50:16.205 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:16.206 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 31 k 2024-03-29T10:50:16.206 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 140 k 2024-03-29T10:50:16.206 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-03-29T10:50:16.366 INFO:teuthology.orchestra.run.smithi112.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 195 kB/s | 31 kB 00:00 2024-03-29T10:50:16.367 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:16.367 INFO:teuthology.orchestra.run.smithi112.stdout:Total 193 kB/s | 31 kB 00:00 2024-03-29T10:50:16.367 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-03-29T10:50:16.371 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-03-29T10:50:16.371 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-03-29T10:50:16.416 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-03-29T10:50:16.416 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-03-29T10:50:16.541 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-03-29T10:50:16.640 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:17.023 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:17.283 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:17.283 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:17.283 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-03-29T10:50:17.283 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:17.283 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:17.284 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:50:17.475 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librados2 2024-03-29T10:50:18.068 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Mar 2024 10:48:20 AM UTC. 2024-03-29T10:50:18.169 INFO:teuthology.orchestra.run.smithi112.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:18.224 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:50:18.225 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-03-29T10:50:18.225 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:50:18.277 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librbd1 2024-03-29T10:50:18.848 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Mar 2024 10:48:20 AM UTC. 2024-03-29T10:50:18.944 INFO:teuthology.orchestra.run.smithi112.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:18.995 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:50:18.996 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-03-29T10:50:18.996 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:50:19.058 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rados 2024-03-29T10:50:19.618 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:59 ago on Fri 29 Mar 2024 10:48:20 AM UTC. 2024-03-29T10:50:19.716 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:19.768 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:50:19.769 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-03-29T10:50:19.769 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:50:19.815 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rgw 2024-03-29T10:50:20.375 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:00 ago on Fri 29 Mar 2024 10:48:20 AM UTC. 2024-03-29T10:50:20.473 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:20.525 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:50:20.526 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-03-29T10:50:20.527 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:50:20.581 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-cephfs 2024-03-29T10:50:21.162 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:01 ago on Fri 29 Mar 2024 10:48:20 AM UTC. 2024-03-29T10:50:21.259 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:21.311 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:50:21.312 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-03-29T10:50:21.312 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:50:21.366 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rbd 2024-03-29T10:50:21.931 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:01 ago on Fri 29 Mar 2024 10:48:20 AM UTC. 2024-03-29T10:50:22.029 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:22.081 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:50:22.082 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-03-29T10:50:22.082 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:50:22.141 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-fuse 2024-03-29T10:50:22.698 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:02 ago on Fri 29 Mar 2024 10:48:20 AM UTC. 2024-03-29T10:50:22.845 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:50:22.846 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:50:22.846 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2024-03-29T10:50:22.846 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:50:22.846 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-03-29T10:50:22.846 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:50:22.847 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:22.847 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-03-29T10:50:22.847 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:50:22.847 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-03-29T10:50:22.847 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:22.847 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 85 k 2024-03-29T10:50:22.847 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 231 k 2024-03-29T10:50:22.847 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-03-29T10:50:23.053 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 415 kB/s | 85 kB 00:00 2024-03-29T10:50:23.053 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:23.054 INFO:teuthology.orchestra.run.smithi112.stdout:Total 412 kB/s | 85 kB 00:00 2024-03-29T10:50:23.054 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-03-29T10:50:23.061 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-03-29T10:50:23.061 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-03-29T10:50:23.117 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-03-29T10:50:23.117 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-03-29T10:50:23.259 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-03-29T10:50:23.367 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:23.785 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:24.085 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:24.085 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:24.085 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-03-29T10:50:24.085 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:24.085 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:24.085 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:50:24.314 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-mirror 2024-03-29T10:50:24.890 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:04 ago on Fri 29 Mar 2024 10:48:20 AM UTC. 2024-03-29T10:50:25.044 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:50:25.045 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:50:25.045 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-03-29T10:50:25.045 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:50:25.045 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-03-29T10:50:25.045 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:50:25.045 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:25.045 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-03-29T10:50:25.045 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:50:25.045 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-03-29T10:50:25.046 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:25.046 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 3.1 M 2024-03-29T10:50:25.046 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 13 M 2024-03-29T10:50:25.046 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-03-29T10:50:25.430 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 8.0 MB/s | 3.1 MB 00:00 2024-03-29T10:50:25.431 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:25.431 INFO:teuthology.orchestra.run.smithi112.stdout:Total 8.0 MB/s | 3.1 MB 00:00 2024-03-29T10:50:25.431 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-03-29T10:50:25.440 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-03-29T10:50:25.441 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-03-29T10:50:25.516 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-03-29T10:50:25.516 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-03-29T10:50:25.946 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-03-29T10:50:26.006 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:26.031 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:26.031 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:50:26.031 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:50:26.031 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:50:26.031 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:50:26.032 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:27.207 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:27.208 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:27.208 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-03-29T10:50:27.208 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:27.208 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:27.208 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:50:27.448 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-nbd 2024-03-29T10:50:28.026 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:08 ago on Fri 29 Mar 2024 10:48:20 AM UTC. 2024-03-29T10:50:28.174 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-03-29T10:50:28.175 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:50:28.175 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2024-03-29T10:50:28.175 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:50:28.175 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-03-29T10:50:28.175 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:50:28.175 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:28.175 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-03-29T10:50:28.176 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-03-29T10:50:28.176 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-03-29T10:50:28.176 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:28.176 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 171 k 2024-03-29T10:50:28.176 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 494 k 2024-03-29T10:50:28.176 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-03-29T10:50:28.858 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 251 kB/s | 171 kB 00:00 2024-03-29T10:50:28.859 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:28.859 INFO:teuthology.orchestra.run.smithi112.stdout:Total 251 kB/s | 171 kB 00:00 2024-03-29T10:50:28.860 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-03-29T10:50:28.867 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-03-29T10:50:28.867 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-03-29T10:50:28.923 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-03-29T10:50:28.923 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-03-29T10:50:29.068 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-03-29T10:50:29.173 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:29.620 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:29.899 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:29.899 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:29.899 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-03-29T10:50:29.899 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:29.899 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:29.899 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-03-29T10:50:30.134 DEBUG:teuthology.parallel:result is None 2024-03-29T10:50:30.135 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:50:30.359 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:50:30.380 INFO:teuthology.orchestra.run.smithi112.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:50:30.381 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:50:30.381 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:50:30.383 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:50:30.521 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:50:30.547 INFO:teuthology.orchestra.run.smithi118.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:50:30.547 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:50:30.548 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:50:30.549 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T10:50:30.550 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-03-29T10:50:30.550 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:50:30.578 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-29T10:50:30.578 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:50:30.620 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T10:50:30.620 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-03-29T10:50:30.620 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:50:30.648 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:50:30.715 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-29T10:50:30.715 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:50:30.746 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:50:30.815 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T10:50:30.815 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-03-29T10:50:30.816 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:50:30.842 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:50:30.909 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-29T10:50:30.909 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:50:30.941 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:50:31.010 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T10:50:31.011 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-03-29T10:50:31.011 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:50:31.038 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:50:31.105 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-29T10:50:31.105 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:50:31.132 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:50:31.202 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T10:50:31.295 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T10:50:31.295 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 777 /var/log/ceph 2024-03-29T10:50:31.297 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 777 /var/log/ceph 2024-03-29T10:50:31.324 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T10:50:31.324 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:50:31.365 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:50:31.391 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T10:50:31.391 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:50:31.431 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:50:31.468 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T10:50:31.468 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T10:50:31.469 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622657', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622657', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi112.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHuBDjejHQ6clU+zE35ohc++KSsObk4DDq5tqpmPWL13rsLQqLqicbLxqFFy7qS/SeaQoaT9UKuFHowzwksXC5w=', 'smithi118.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJLpc8+kcufOIZ99A3ZlOO/UURp3bM65vBveLNBHcmJbrtsmZRV2it8LQ7naivtFd3Kr+YdOsKlHVhkdpc5FJHU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T10:50:31.469 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:50:31.498 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:50:31.513 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:50:31.553 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:50:31.579 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-03-29T10:50:31.580 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:50:31.633 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:50:31.633 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_1 2024-03-29T10:50:31.687 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:50:31.687 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:50:31.687 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T10:50:31.687 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:50:31.687 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:50:31.688 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-03-29 10:50:26.307146363 +0000 2024-03-29T10:50:31.688 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-03-29 10:47:35.177644166 +0000 2024-03-29T10:50:31.688 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-03-29 10:47:35.177644166 +0000 2024-03-29T10:50:31.688 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-03-29 10:47:35.177644166 +0000 2024-03-29T10:50:31.688 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:50:31.753 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-03-29T10:50:31.753 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-03-29T10:50:31.753 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000259224 s, 2.0 MB/s 2024-03-29T10:50:31.754 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:50:31.810 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_2 2024-03-29T10:50:31.865 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:50:31.865 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:50:31.865 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T10:50:31.865 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:50:31.865 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:50:31.865 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-03-29 10:50:26.307146363 +0000 2024-03-29T10:50:31.866 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-03-29 10:47:35.515635734 +0000 2024-03-29T10:50:31.866 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-03-29 10:47:35.515635734 +0000 2024-03-29T10:50:31.866 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-03-29 10:47:35.515635734 +0000 2024-03-29T10:50:31.866 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:50:31.930 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-03-29T10:50:31.931 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-03-29T10:50:31.931 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000209668 s, 2.4 MB/s 2024-03-29T10:50:31.932 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:50:31.988 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_3 2024-03-29T10:50:32.043 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:50:32.044 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:50:32.044 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T10:50:32.044 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:50:32.044 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:50:32.044 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-03-29 10:50:26.307146363 +0000 2024-03-29T10:50:32.044 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-03-29 10:47:35.867626954 +0000 2024-03-29T10:50:32.044 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-03-29 10:47:35.867626954 +0000 2024-03-29T10:50:32.044 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-03-29 10:47:35.867626954 +0000 2024-03-29T10:50:32.044 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:50:32.108 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-03-29T10:50:32.133 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-03-29T10:50:32.133 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000195046 s, 2.6 MB/s 2024-03-29T10:50:32.134 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:50:32.190 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_4 2024-03-29T10:50:32.245 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:50:32.245 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:50:32.245 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T10:50:32.245 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:50:32.246 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:50:32.246 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-03-29 10:50:26.308146338 +0000 2024-03-29T10:50:32.246 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-03-29 10:47:36.220618148 +0000 2024-03-29T10:50:32.246 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-03-29 10:47:36.220618148 +0000 2024-03-29T10:50:32.246 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-03-29 10:47:36.220618148 +0000 2024-03-29T10:50:32.246 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:50:32.311 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-03-29T10:50:32.311 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-03-29T10:50:32.311 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000220673 s, 2.3 MB/s 2024-03-29T10:50:32.312 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:50:32.368 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:50:32.368 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-29T10:50:32.368 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:50:32.384 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:50:32.384 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_1 2024-03-29T10:50:32.439 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:50:32.439 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:50:32.439 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T10:50:32.439 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:50:32.439 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:50:32.439 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-03-29 10:50:12.396348117 +0000 2024-03-29T10:50:32.439 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-03-29 10:47:35.134032532 +0000 2024-03-29T10:50:32.439 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-03-29 10:47:35.134032532 +0000 2024-03-29T10:50:32.439 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-03-29 10:47:35.134032532 +0000 2024-03-29T10:50:32.440 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:50:32.505 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-03-29T10:50:32.505 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-03-29T10:50:32.505 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 8.9714e-05 s, 5.7 MB/s 2024-03-29T10:50:32.506 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:50:32.562 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_2 2024-03-29T10:50:32.617 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:50:32.617 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:50:32.617 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T10:50:32.617 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:50:32.617 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:50:32.617 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-03-29 10:50:12.397348095 +0000 2024-03-29T10:50:32.617 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-03-29 10:47:35.477024613 +0000 2024-03-29T10:50:32.617 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-03-29 10:47:35.477024613 +0000 2024-03-29T10:50:32.617 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-03-29 10:47:35.477024613 +0000 2024-03-29T10:50:32.617 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:50:32.682 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-03-29T10:50:32.682 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-03-29T10:50:32.682 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 9.8615e-05 s, 5.2 MB/s 2024-03-29T10:50:32.683 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:50:32.739 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_3 2024-03-29T10:50:32.793 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:50:32.794 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:50:32.794 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-29T10:50:32.794 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:50:32.794 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:50:32.794 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-03-29 10:50:12.397348095 +0000 2024-03-29T10:50:32.794 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-03-29 10:47:35.807016995 +0000 2024-03-29T10:50:32.794 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-03-29 10:47:35.807016995 +0000 2024-03-29T10:50:32.795 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-03-29 10:47:35.807016995 +0000 2024-03-29T10:50:32.795 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:50:32.859 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-03-29T10:50:32.859 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-03-29T10:50:32.859 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 9.5456e-05 s, 5.4 MB/s 2024-03-29T10:50:32.860 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:50:32.915 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_4 2024-03-29T10:50:32.970 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:50:32.971 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:50:32.971 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T10:50:32.971 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:50:32.971 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:50:32.971 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-03-29 10:50:12.397348095 +0000 2024-03-29T10:50:32.971 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-03-29 10:47:36.342004643 +0000 2024-03-29T10:50:32.971 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-03-29 10:47:36.342004643 +0000 2024-03-29T10:50:32.971 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-03-29 10:47:36.342004643 +0000 2024-03-29T10:50:32.972 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:50:33.035 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-03-29T10:50:33.035 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-03-29T10:50:33.035 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000100605 s, 5.1 MB/s 2024-03-29T10:50:33.036 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:50:33.092 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:50:33.092 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi118.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:50:33.092 INFO:tasks.ceph:Generating config... 2024-03-29T10:50:33.094 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T10:50:33.094 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T10:50:33.094 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T10:50:33.094 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T10:50:33.094 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T10:50:33.095 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T10:50:33.095 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T10:50:33.095 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T10:50:33.095 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T10:50:33.095 INFO:tasks.ceph:[global] ms type = async 2024-03-29T10:50:33.095 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T10:50:33.095 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T10:50:33.095 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T10:50:33.095 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T10:50:33.095 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T10:50:33.095 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T10:50:33.096 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T10:50:33.096 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T10:50:33.096 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T10:50:33.096 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T10:50:33.096 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T10:50:33.096 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T10:50:33.096 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T10:50:33.096 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T10:50:33.096 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T10:50:33.096 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T10:50:33.096 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T10:50:33.096 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T10:50:33.096 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T10:50:33.097 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T10:50:33.097 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T10:50:33.097 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T10:50:33.097 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T10:50:33.097 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T10:50:33.097 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T10:50:33.097 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T10:50:33.097 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T10:50:33.097 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T10:50:33.097 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T10:50:33.097 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T10:50:33.097 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T10:50:33.097 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T10:50:33.097 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T10:50:33.098 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T10:50:33.098 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T10:50:33.098 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T10:50:33.098 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T10:50:33.098 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T10:50:33.098 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T10:50:33.098 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T10:50:33.146 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T10:50:33.149 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T10:50:33.194 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:50:33.221 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.112:6789]'), ('mon.c', '[v1:172.21.15.112:6790]'), ('mon.b', '[v1:172.21.15.118:6789]')] 2024-03-29T10:50:33.222 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.112:6789],[v1:172.21.15.112:6790],[v1:172.21.15.118:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T10:50:33.222 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-03-29T10:50:33.222 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:50:33.276 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.112:6789]' --addv c '[v1:172.21.15.112:6790]' --addv b '[v1:172.21.15.118:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:50:33.362 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:50:33.362 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: generated fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:50:33.362 INFO:teuthology.orchestra.run.smithi112.stdout:setting min_mon_release = quincy 2024-03-29T10:50:33.362 INFO:teuthology.orchestra.run.smithi112.stdout:epoch 0 2024-03-29T10:50:33.363 INFO:teuthology.orchestra.run.smithi112.stdout:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:50:33.363 INFO:teuthology.orchestra.run.smithi112.stdout:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:50:33.363 INFO:teuthology.orchestra.run.smithi112.stdout:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:50:33.363 INFO:teuthology.orchestra.run.smithi112.stdout:min_mon_release 17 (quincy) 2024-03-29T10:50:33.363 INFO:teuthology.orchestra.run.smithi112.stdout:election_strategy: 1 2024-03-29T10:50:33.363 INFO:teuthology.orchestra.run.smithi112.stdout:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:50:33.363 INFO:teuthology.orchestra.run.smithi112.stdout:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:50:33.363 INFO:teuthology.orchestra.run.smithi112.stdout:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:50:33.363 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T10:50:33.364 DEBUG:teuthology.orchestra.run.smithi112:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:50:33.379 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a3151ab9-9e71-4afa-ac98-17b78e4993d2... 2024-03-29T10:50:33.380 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:50:33.422 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:50:33.470 INFO:teuthology.orchestra.run.smithi112.stdout:[global] 2024-03-29T10:50:33.470 INFO:teuthology.orchestra.run.smithi112.stdout: chdir = "" 2024-03-29T10:50:33.470 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:50:33.470 INFO:teuthology.orchestra.run.smithi112.stdout: auth supported = cephx 2024-03-29T10:50:33.471 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.471 INFO:teuthology.orchestra.run.smithi112.stdout: filestore xattr use omap = true 2024-03-29T10:50:33.471 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.471 INFO:teuthology.orchestra.run.smithi112.stdout: mon clock drift allowed = 1.000 2024-03-29T10:50:33.471 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.471 INFO:teuthology.orchestra.run.smithi112.stdout: osd crush chooseleaf type = 0 2024-03-29T10:50:33.471 INFO:teuthology.orchestra.run.smithi112.stdout: auth debug = true 2024-03-29T10:50:33.471 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.471 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on old message = true 2024-03-29T10:50:33.471 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on bug = true 2024-03-29T10:50:33.471 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.471 INFO:teuthology.orchestra.run.smithi112.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:50:33.471 INFO:teuthology.orchestra.run.smithi112.stdout: mon pg warn max object skew = 0 2024-03-29T10:50:33.472 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.472 INFO:teuthology.orchestra.run.smithi112.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:50:33.472 INFO:teuthology.orchestra.run.smithi112.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:50:33.472 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.472 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default size = 2 2024-03-29T10:50:33.472 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.472 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow primary affinity = true 2024-03-29T10:50:33.472 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow pg remap = true 2024-03-29T10:50:33.472 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:50:33.472 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:50:33.472 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on no sortbitwise = false 2024-03-29T10:50:33.472 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:50:33.472 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on too few osds = false 2024-03-29T10:50:33.472 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:50:33.473 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:50:33.473 INFO:teuthology.orchestra.run.smithi112.stdout: mon_allow_pool_size_one = true 2024-03-29T10:50:33.473 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.473 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:50:33.473 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.473 INFO:teuthology.orchestra.run.smithi112.stdout: osd default data pool replay window = 5 2024-03-29T10:50:33.473 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.473 INFO:teuthology.orchestra.run.smithi112.stdout: mon allow pool delete = true 2024-03-29T10:50:33.473 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.473 INFO:teuthology.orchestra.run.smithi112.stdout: mon cluster log file level = debug 2024-03-29T10:50:33.473 INFO:teuthology.orchestra.run.smithi112.stdout: debug asserts on shutdown = true 2024-03-29T10:50:33.473 INFO:teuthology.orchestra.run.smithi112.stdout: mon health detail to clog = false 2024-03-29T10:50:33.473 INFO:teuthology.orchestra.run.smithi112.stdout: mon host = "[v1:172.21.15.112:6789],[v1:172.21.15.112:6790],[v1:172.21.15.118:6789]" 2024-03-29T10:50:33.473 INFO:teuthology.orchestra.run.smithi112.stdout: lockdep = True 2024-03-29T10:50:33.474 INFO:teuthology.orchestra.run.smithi112.stdout: mon client directed command retry = 5 2024-03-29T10:50:33.474 INFO:teuthology.orchestra.run.smithi112.stdout: mon election default strategy = 1 2024-03-29T10:50:33.474 INFO:teuthology.orchestra.run.smithi112.stdout: ms bind msgr2 = False 2024-03-29T10:50:33.474 INFO:teuthology.orchestra.run.smithi112.stdout: ms inject socket failures = 5000 2024-03-29T10:50:33.474 INFO:teuthology.orchestra.run.smithi112.stdout: ms type = async 2024-03-29T10:50:33.474 INFO:teuthology.orchestra.run.smithi112.stdout: fsid = a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:50:33.474 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.474 INFO:teuthology.orchestra.run.smithi112.stdout:[osd] 2024-03-29T10:50:33.474 INFO:teuthology.orchestra.run.smithi112.stdout: osd journal size = 100 2024-03-29T10:50:33.474 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.474 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub load threshold = 5.0 2024-03-29T10:50:33.474 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub max interval = 600 2024-03-29T10:50:33.474 INFO:teuthology.orchestra.run.smithi112.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:50:33.474 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.475 INFO:teuthology.orchestra.run.smithi112.stdout: osd recover clone overlap = true 2024-03-29T10:50:33.475 INFO:teuthology.orchestra.run.smithi112.stdout: osd recovery max chunk = 1048576 2024-03-29T10:50:33.475 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.475 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug shutdown = true 2024-03-29T10:50:33.475 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug op order = true 2024-03-29T10:50:33.475 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug verify stray on activate = true 2024-03-29T10:50:33.475 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug trim objects = true 2024-03-29T10:50:33.475 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.475 INFO:teuthology.orchestra.run.smithi112.stdout: osd open classes on start = true 2024-03-29T10:50:33.475 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug pg log writeout = true 2024-03-29T10:50:33.475 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.475 INFO:teuthology.orchestra.run.smithi112.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:50:33.475 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.475 INFO:teuthology.orchestra.run.smithi112.stdout: osd map max advance = 10 2024-03-29T10:50:33.476 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.476 INFO:teuthology.orchestra.run.smithi112.stdout: journal zero on create = true 2024-03-29T10:50:33.476 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.476 INFO:teuthology.orchestra.run.smithi112.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:50:33.476 INFO:teuthology.orchestra.run.smithi112.stdout: filestore apply finisher threads = 3 2024-03-29T10:50:33.476 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.476 INFO:teuthology.orchestra.run.smithi112.stdout: bdev debug aio = true 2024-03-29T10:50:33.476 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug misdirected ops = true 2024-03-29T10:50:33.476 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore block size = 96636764160 2024-03-29T10:50:33.476 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore compression algorithm = zstd 2024-03-29T10:50:33.476 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore compression mode = aggressive 2024-03-29T10:50:33.476 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore fsck on mount = True 2024-03-29T10:50:33.476 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore zero block detection = True 2024-03-29T10:50:33.476 INFO:teuthology.orchestra.run.smithi112.stdout: debug bluefs = 20 2024-03-29T10:50:33.476 INFO:teuthology.orchestra.run.smithi112.stdout: debug bluestore = 20 2024-03-29T10:50:33.477 INFO:teuthology.orchestra.run.smithi112.stdout: debug monc = 20 2024-03-29T10:50:33.477 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-03-29T10:50:33.477 INFO:teuthology.orchestra.run.smithi112.stdout: debug osd = 20 2024-03-29T10:50:33.477 INFO:teuthology.orchestra.run.smithi112.stdout: debug rocksdb = 10 2024-03-29T10:50:33.477 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:50:33.477 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd full ratio = 0.9 2024-03-29T10:50:33.477 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:50:33.477 INFO:teuthology.orchestra.run.smithi112.stdout: osd class default list = * 2024-03-29T10:50:33.477 INFO:teuthology.orchestra.run.smithi112.stdout: osd class load list = * 2024-03-29T10:50:33.477 INFO:teuthology.orchestra.run.smithi112.stdout: osd client watch timeout = 120 2024-03-29T10:50:33.477 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug verify cached snaps = True 2024-03-29T10:50:33.477 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug verify missing on start = True 2024-03-29T10:50:33.477 INFO:teuthology.orchestra.run.smithi112.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:50:33.477 INFO:teuthology.orchestra.run.smithi112.stdout: osd mclock override recovery settings = True 2024-03-29T10:50:33.477 INFO:teuthology.orchestra.run.smithi112.stdout: osd objectstore = bluestore 2024-03-29T10:50:33.478 INFO:teuthology.orchestra.run.smithi112.stdout: osd op queue = debug_random 2024-03-29T10:50:33.478 INFO:teuthology.orchestra.run.smithi112.stdout: osd op queue cut off = debug_random 2024-03-29T10:50:33.478 INFO:teuthology.orchestra.run.smithi112.stdout: osd shutdown pgref assert = True 2024-03-29T10:50:33.478 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.478 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr] 2024-03-29T10:50:33.478 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-03-29T10:50:33.478 INFO:teuthology.orchestra.run.smithi112.stdout: debug mgr = 20 2024-03-29T10:50:33.478 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2024-03-29T10:50:33.478 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2024-03-29T10:50:33.478 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:50:33.478 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:50:33.478 INFO:teuthology.orchestra.run.smithi112.stdout: mgr/telemetry/nag = false 2024-03-29T10:50:33.478 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.479 INFO:teuthology.orchestra.run.smithi112.stdout:[mon] 2024-03-29T10:50:33.479 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-03-29T10:50:33.479 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2024-03-29T10:50:33.479 INFO:teuthology.orchestra.run.smithi112.stdout: debug paxos = 20 2024-03-29T10:50:33.479 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2024-03-29T10:50:33.479 INFO:teuthology.orchestra.run.smithi112.stdout: mon data avail warn = 5 2024-03-29T10:50:33.479 INFO:teuthology.orchestra.run.smithi112.stdout: mon mgr mkfs grace = 240 2024-03-29T10:50:33.479 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:50:33.479 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd reporter subtree level = osd 2024-03-29T10:50:33.479 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd prime pg temp = true 2024-03-29T10:50:33.479 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:50:33.479 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.479 INFO:teuthology.orchestra.run.smithi112.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:50:33.479 INFO:teuthology.orchestra.run.smithi112.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:50:33.480 INFO:teuthology.orchestra.run.smithi112.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:50:33.480 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.480 INFO:teuthology.orchestra.run.smithi112.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:50:33.480 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:50:33.480 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:50:33.480 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.480 INFO:teuthology.orchestra.run.smithi112.stdout: # 1m isn't quite enough 2024-03-29T10:50:33.480 INFO:teuthology.orchestra.run.smithi112.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:50:33.480 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.480 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:50:33.480 INFO:teuthology.orchestra.run.smithi112.stdout: mon min osdmap epochs = 50 2024-03-29T10:50:33.480 INFO:teuthology.orchestra.run.smithi112.stdout: mon osdmap full prune interval = 2 2024-03-29T10:50:33.480 INFO:teuthology.orchestra.run.smithi112.stdout: mon osdmap full prune min = 15 2024-03-29T10:50:33.480 INFO:teuthology.orchestra.run.smithi112.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:50:33.480 INFO:teuthology.orchestra.run.smithi112.stdout: mon scrub interval = 300 2024-03-29T10:50:33.481 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on pool no app = False 2024-03-29T10:50:33.481 INFO:teuthology.orchestra.run.smithi112.stdout: paxos service trim min = 10 2024-03-29T10:50:33.481 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:50:33.481 INFO:teuthology.orchestra.run.smithi112.stdout:[client] 2024-03-29T10:50:33.481 INFO:teuthology.orchestra.run.smithi112.stdout: rgw cache enabled = true 2024-03-29T10:50:33.481 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable ops log = true 2024-03-29T10:50:33.481 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable usage log = true 2024-03-29T10:50:33.481 INFO:teuthology.orchestra.run.smithi112.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:50:33.481 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:50:33.481 INFO:teuthology.orchestra.run.smithi112.stdout: debug monc = 20 2024-03-29T10:50:33.481 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-03-29T10:50:33.481 INFO:teuthology.orchestra.run.smithi112.stdout: debug objecter = 20 2024-03-29T10:50:33.481 INFO:teuthology.orchestra.run.smithi112.stdout: debug rados = 20 2024-03-29T10:50:33.481 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.a] 2024-03-29T10:50:33.481 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.c] 2024-03-29T10:50:33.482 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.b] 2024-03-29T10:50:33.482 INFO:teuthology.orchestra.run.smithi118.stdout:[global] 2024-03-29T10:50:33.482 INFO:teuthology.orchestra.run.smithi118.stdout: chdir = "" 2024-03-29T10:50:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:50:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: auth supported = cephx 2024-03-29T10:50:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: filestore xattr use omap = true 2024-03-29T10:50:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: mon clock drift allowed = 1.000 2024-03-29T10:50:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: osd crush chooseleaf type = 0 2024-03-29T10:50:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: auth debug = true 2024-03-29T10:50:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on old message = true 2024-03-29T10:50:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on bug = true 2024-03-29T10:50:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.483 INFO:teuthology.orchestra.run.smithi118.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:50:33.484 INFO:teuthology.orchestra.run.smithi118.stdout: mon pg warn max object skew = 0 2024-03-29T10:50:33.484 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.484 INFO:teuthology.orchestra.run.smithi118.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:50:33.484 INFO:teuthology.orchestra.run.smithi118.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:50:33.484 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.484 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default size = 2 2024-03-29T10:50:33.484 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.484 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow primary affinity = true 2024-03-29T10:50:33.484 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow pg remap = true 2024-03-29T10:50:33.484 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:50:33.484 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:50:33.484 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on no sortbitwise = false 2024-03-29T10:50:33.484 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:50:33.484 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on too few osds = false 2024-03-29T10:50:33.485 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:50:33.485 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:50:33.485 INFO:teuthology.orchestra.run.smithi118.stdout: mon_allow_pool_size_one = true 2024-03-29T10:50:33.485 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.485 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:50:33.485 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.485 INFO:teuthology.orchestra.run.smithi118.stdout: osd default data pool replay window = 5 2024-03-29T10:50:33.485 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.485 INFO:teuthology.orchestra.run.smithi118.stdout: mon allow pool delete = true 2024-03-29T10:50:33.485 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.485 INFO:teuthology.orchestra.run.smithi118.stdout: mon cluster log file level = debug 2024-03-29T10:50:33.485 INFO:teuthology.orchestra.run.smithi118.stdout: debug asserts on shutdown = true 2024-03-29T10:50:33.486 INFO:teuthology.orchestra.run.smithi118.stdout: mon health detail to clog = false 2024-03-29T10:50:33.486 INFO:teuthology.orchestra.run.smithi118.stdout: mon host = "[v1:172.21.15.112:6789],[v1:172.21.15.112:6790],[v1:172.21.15.118:6789]" 2024-03-29T10:50:33.486 INFO:teuthology.orchestra.run.smithi118.stdout: lockdep = True 2024-03-29T10:50:33.486 INFO:teuthology.orchestra.run.smithi118.stdout: mon client directed command retry = 5 2024-03-29T10:50:33.486 INFO:teuthology.orchestra.run.smithi118.stdout: mon election default strategy = 1 2024-03-29T10:50:33.486 INFO:teuthology.orchestra.run.smithi118.stdout: ms bind msgr2 = False 2024-03-29T10:50:33.486 INFO:teuthology.orchestra.run.smithi118.stdout: ms inject socket failures = 5000 2024-03-29T10:50:33.486 INFO:teuthology.orchestra.run.smithi118.stdout: ms type = async 2024-03-29T10:50:33.486 INFO:teuthology.orchestra.run.smithi118.stdout: fsid = a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:50:33.486 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.487 INFO:teuthology.orchestra.run.smithi118.stdout:[osd] 2024-03-29T10:50:33.487 INFO:teuthology.orchestra.run.smithi118.stdout: osd journal size = 100 2024-03-29T10:50:33.487 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.487 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub load threshold = 5.0 2024-03-29T10:50:33.487 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub max interval = 600 2024-03-29T10:50:33.487 INFO:teuthology.orchestra.run.smithi118.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:50:33.487 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.487 INFO:teuthology.orchestra.run.smithi118.stdout: osd recover clone overlap = true 2024-03-29T10:50:33.487 INFO:teuthology.orchestra.run.smithi118.stdout: osd recovery max chunk = 1048576 2024-03-29T10:50:33.487 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug shutdown = true 2024-03-29T10:50:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug op order = true 2024-03-29T10:50:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify stray on activate = true 2024-03-29T10:50:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug trim objects = true 2024-03-29T10:50:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: osd open classes on start = true 2024-03-29T10:50:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug pg log writeout = true 2024-03-29T10:50:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:50:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.488 INFO:teuthology.orchestra.run.smithi118.stdout: osd map max advance = 10 2024-03-29T10:50:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: journal zero on create = true 2024-03-29T10:50:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:50:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: filestore apply finisher threads = 3 2024-03-29T10:50:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: bdev debug aio = true 2024-03-29T10:50:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug misdirected ops = true 2024-03-29T10:50:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore block size = 96636764160 2024-03-29T10:50:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore compression algorithm = zstd 2024-03-29T10:50:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore compression mode = aggressive 2024-03-29T10:50:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore fsck on mount = True 2024-03-29T10:50:33.490 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore zero block detection = True 2024-03-29T10:50:33.490 INFO:teuthology.orchestra.run.smithi118.stdout: debug bluefs = 20 2024-03-29T10:50:33.490 INFO:teuthology.orchestra.run.smithi118.stdout: debug bluestore = 20 2024-03-29T10:50:33.490 INFO:teuthology.orchestra.run.smithi118.stdout: debug monc = 20 2024-03-29T10:50:33.490 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2024-03-29T10:50:33.490 INFO:teuthology.orchestra.run.smithi118.stdout: debug osd = 20 2024-03-29T10:50:33.490 INFO:teuthology.orchestra.run.smithi118.stdout: debug rocksdb = 10 2024-03-29T10:50:33.490 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:50:33.490 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd full ratio = 0.9 2024-03-29T10:50:33.490 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:50:33.491 INFO:teuthology.orchestra.run.smithi118.stdout: osd class default list = * 2024-03-29T10:50:33.491 INFO:teuthology.orchestra.run.smithi118.stdout: osd class load list = * 2024-03-29T10:50:33.491 INFO:teuthology.orchestra.run.smithi118.stdout: osd client watch timeout = 120 2024-03-29T10:50:33.491 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify cached snaps = True 2024-03-29T10:50:33.491 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify missing on start = True 2024-03-29T10:50:33.491 INFO:teuthology.orchestra.run.smithi118.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:50:33.491 INFO:teuthology.orchestra.run.smithi118.stdout: osd mclock override recovery settings = True 2024-03-29T10:50:33.491 INFO:teuthology.orchestra.run.smithi118.stdout: osd objectstore = bluestore 2024-03-29T10:50:33.491 INFO:teuthology.orchestra.run.smithi118.stdout: osd op queue = debug_random 2024-03-29T10:50:33.491 INFO:teuthology.orchestra.run.smithi118.stdout: osd op queue cut off = debug_random 2024-03-29T10:50:33.491 INFO:teuthology.orchestra.run.smithi118.stdout: osd shutdown pgref assert = True 2024-03-29T10:50:33.491 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.491 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr] 2024-03-29T10:50:33.492 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2024-03-29T10:50:33.492 INFO:teuthology.orchestra.run.smithi118.stdout: debug mgr = 20 2024-03-29T10:50:33.492 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2024-03-29T10:50:33.492 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2024-03-29T10:50:33.492 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:50:33.492 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:50:33.492 INFO:teuthology.orchestra.run.smithi118.stdout: mgr/telemetry/nag = false 2024-03-29T10:50:33.492 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.492 INFO:teuthology.orchestra.run.smithi118.stdout:[mon] 2024-03-29T10:50:33.492 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2024-03-29T10:50:33.492 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2024-03-29T10:50:33.493 INFO:teuthology.orchestra.run.smithi118.stdout: debug paxos = 20 2024-03-29T10:50:33.493 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2024-03-29T10:50:33.493 INFO:teuthology.orchestra.run.smithi118.stdout: mon data avail warn = 5 2024-03-29T10:50:33.493 INFO:teuthology.orchestra.run.smithi118.stdout: mon mgr mkfs grace = 240 2024-03-29T10:50:33.493 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:50:33.493 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd reporter subtree level = osd 2024-03-29T10:50:33.493 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd prime pg temp = true 2024-03-29T10:50:33.493 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:50:33.493 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.493 INFO:teuthology.orchestra.run.smithi118.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:50:33.493 INFO:teuthology.orchestra.run.smithi118.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:50:33.493 INFO:teuthology.orchestra.run.smithi118.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:50:33.494 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.494 INFO:teuthology.orchestra.run.smithi118.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:50:33.494 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:50:33.494 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:50:33.494 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.494 INFO:teuthology.orchestra.run.smithi118.stdout: # 1m isn't quite enough 2024-03-29T10:50:33.494 INFO:teuthology.orchestra.run.smithi118.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:50:33.494 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.494 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:50:33.494 INFO:teuthology.orchestra.run.smithi118.stdout: mon min osdmap epochs = 50 2024-03-29T10:50:33.494 INFO:teuthology.orchestra.run.smithi118.stdout: mon osdmap full prune interval = 2 2024-03-29T10:50:33.494 INFO:teuthology.orchestra.run.smithi118.stdout: mon osdmap full prune min = 15 2024-03-29T10:50:33.494 INFO:teuthology.orchestra.run.smithi118.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:50:33.494 INFO:teuthology.orchestra.run.smithi118.stdout: mon scrub interval = 300 2024-03-29T10:50:33.495 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on pool no app = False 2024-03-29T10:50:33.495 INFO:teuthology.orchestra.run.smithi118.stdout: paxos service trim min = 10 2024-03-29T10:50:33.495 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-29T10:50:33.495 INFO:teuthology.orchestra.run.smithi118.stdout:[client] 2024-03-29T10:50:33.495 INFO:teuthology.orchestra.run.smithi118.stdout: rgw cache enabled = true 2024-03-29T10:50:33.495 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable ops log = true 2024-03-29T10:50:33.495 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable usage log = true 2024-03-29T10:50:33.495 INFO:teuthology.orchestra.run.smithi118.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:50:33.495 INFO:teuthology.orchestra.run.smithi118.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:50:33.495 INFO:teuthology.orchestra.run.smithi118.stdout: debug monc = 20 2024-03-29T10:50:33.495 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2024-03-29T10:50:33.495 INFO:teuthology.orchestra.run.smithi118.stdout: debug objecter = 20 2024-03-29T10:50:33.495 INFO:teuthology.orchestra.run.smithi118.stdout: debug rados = 20 2024-03-29T10:50:33.495 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.a] 2024-03-29T10:50:33.496 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.c] 2024-03-29T10:50:33.496 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.b] 2024-03-29T10:50:33.498 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T10:50:33.498 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T10:50:33.592 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T10:50:33.592 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-03-29T10:50:33.593 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T10:50:33.607 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-03-29T10:50:33.607 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T10:50:33.662 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi112.front.sepia.ceph.com 2024-03-29T10:50:33.663 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-03-29T10:50:33.663 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:50:33.663 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:50:33.741 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-03-29T10:50:33.742 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:50:33.797 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi118.front.sepia.ceph.com 2024-03-29T10:50:33.797 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-29T10:50:33.797 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:50:33.797 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:50:33.837 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-29T10:50:33.837 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:50:33.891 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T10:50:33.892 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T10:50:33.892 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:50:33.946 INFO:teuthology.orchestra.run.smithi112.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:50:33.948 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:50:34.032 INFO:teuthology.orchestra.run.smithi118.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:50:34.034 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T10:50:34.034 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T10:50:34.035 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T10:50:34.077 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T10:50:34.092 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T10:50:34.134 INFO:teuthology.orchestra.run.smithi118.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T10:50:34.149 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T10:50:34.149 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi118.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:50:34.149 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T10:50:34.175 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:50:34.175 INFO:tasks.ceph:role: osd.0 2024-03-29T10:50:34.175 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi112.front.sepia.ceph.com 2024-03-29T10:50:34.175 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:50:34.249 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:50:34.249 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:50:34.250 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:50:34.250 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:50:34.250 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:50:34.250 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:50:34.250 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:50:34.250 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:50:34.250 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:50:34.250 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:50:34.752 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2024-03-29T10:50:34.754 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2024-03-29T10:50:34.754 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T10:50:34.896 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T10:50:34.930 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T10:50:34.998 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:50:34.998 INFO:tasks.ceph:role: osd.1 2024-03-29T10:50:34.998 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi112.front.sepia.ceph.com 2024-03-29T10:50:34.998 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:50:35.073 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:50:35.073 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:50:35.073 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:50:35.073 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:50:35.073 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:50:35.073 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:50:35.073 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:50:35.073 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:50:35.073 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:50:35.073 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:50:35.525 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2024-03-29T10:50:35.527 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2024-03-29T10:50:35.527 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T10:50:35.623 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T10:50:35.653 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T10:50:35.721 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:50:35.721 INFO:tasks.ceph:role: osd.2 2024-03-29T10:50:35.721 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi112.front.sepia.ceph.com 2024-03-29T10:50:35.721 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:50:35.795 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:50:35.796 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:50:35.796 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:50:35.796 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:50:35.796 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:50:35.796 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:50:35.796 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:50:35.796 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:50:35.796 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:50:35.796 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:50:36.285 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2024-03-29T10:50:36.287 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2024-03-29T10:50:36.287 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T10:50:36.440 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T10:50:36.472 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T10:50:36.540 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:50:36.540 INFO:tasks.ceph:role: osd.3 2024-03-29T10:50:36.540 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi112.front.sepia.ceph.com 2024-03-29T10:50:36.540 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:50:36.614 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:50:36.614 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:50:36.614 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:50:36.614 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:50:36.614 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:50:36.614 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:50:36.614 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:50:36.614 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:50:36.614 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:50:36.614 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:50:37.122 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2024-03-29T10:50:37.125 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2024-03-29T10:50:37.125 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T10:50:37.240 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T10:50:37.276 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:50:37.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:50:37.389+0000 7fef48211640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T10:50:37.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:50:37.389+0000 7fef48211640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T10:50:37.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:50:37.389+0000 7fef48211640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:50:37.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:50:37.389+0000 7fef48211640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:50:37.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:50:37.389+0000 7fef48211640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:50:37.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:50:37.391+0000 7fef48211640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T10:50:40.214 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T10:50:40.245 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:50:40.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:50:40.349+0000 7f59aca10640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T10:50:40.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:50:40.349+0000 7f59aca10640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T10:50:40.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:50:40.349+0000 7f59aca10640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:50:40.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:50:40.349+0000 7f59aca10640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:50:40.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:50:40.349+0000 7f59aca10640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:50:40.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:50:40.350+0000 7f59aca10640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T10:50:43.181 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T10:50:43.211 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:50:43.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:50:43.320+0000 7f1eefa0e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T10:50:43.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:50:43.320+0000 7f1eefa0e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T10:50:43.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:50:43.320+0000 7f1eefa0e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:50:43.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:50:43.320+0000 7f1eefa0e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:50:43.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:50:43.320+0000 7f1eefa0e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:50:43.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:50:43.321+0000 7f1eefa0e640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T10:50:46.142 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T10:50:46.170 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:50:46.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:50:46.277+0000 7f2d78499640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T10:50:46.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:50:46.277+0000 7f2d78499640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T10:50:46.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:50:46.277+0000 7f2d78499640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:50:46.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:50:46.277+0000 7f2d78499640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:50:46.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:50:46.277+0000 7f2d78499640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:50:46.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:50:46.277+0000 7f2d78499640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T10:50:49.111 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T10:50:49.571 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T10:50:49.604 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:50:49.604 INFO:tasks.ceph:role: osd.4 2024-03-29T10:50:49.604 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi118.front.sepia.ceph.com 2024-03-29T10:50:49.604 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:50:49.679 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:50:49.679 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:50:49.679 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:50:49.680 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:50:49.680 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:50:49.680 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:50:49.680 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:50:49.680 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:50:49.680 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:50:49.680 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:50:49.711 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2024-03-29T10:50:49.713 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-03-29T10:50:49.714 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T10:50:49.794 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T10:50:49.873 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T10:50:49.952 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:50:49.952 INFO:tasks.ceph:role: osd.5 2024-03-29T10:50:49.953 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi118.front.sepia.ceph.com 2024-03-29T10:50:49.953 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:50:50.023 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:50:50.023 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:50:50.023 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:50:50.023 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:50:50.023 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:50:50.023 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:50:50.023 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:50:50.024 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:50:50.024 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:50:50.024 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:50:50.055 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2024-03-29T10:50:50.057 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-03-29T10:50:50.057 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T10:50:50.137 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T10:50:50.211 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T10:50:50.282 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:50:50.283 INFO:tasks.ceph:role: osd.6 2024-03-29T10:50:50.283 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi118.front.sepia.ceph.com 2024-03-29T10:50:50.283 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:50:50.356 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:50:50.357 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:50:50.357 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:50:50.357 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:50:50.357 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:50:50.357 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:50:50.357 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:50:50.357 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:50:50.357 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:50:50.357 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:50:50.389 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2024-03-29T10:50:50.391 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-03-29T10:50:50.391 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T10:50:50.464 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T10:50:50.536 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T10:50:50.605 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:50:50.605 INFO:tasks.ceph:role: osd.7 2024-03-29T10:50:50.605 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi118.front.sepia.ceph.com 2024-03-29T10:50:50.606 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:50:50.673 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:50:50.674 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:50:50.674 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:50:50.674 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:50:50.674 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:50:50.674 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:50:50.674 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:50:50.674 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:50:50.674 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:50:50.675 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:50:50.706 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2024-03-29T10:50:50.707 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-03-29T10:50:50.708 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T10:50:50.780 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T10:50:50.855 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:50:50.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:50:50.967+0000 7fd3a3614640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T10:50:50.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:50:50.967+0000 7fd3a3614640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T10:50:50.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:50:50.967+0000 7fd3a3614640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:50:50.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:50:50.967+0000 7fd3a3614640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:50:50.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:50:50.967+0000 7fd3a3614640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:50:50.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:50:50.967+0000 7fd3a3614640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T10:50:53.798 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T10:50:53.830 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:50:53.923 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:50:53.941+0000 7f387ecc8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T10:50:53.923 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:50:53.941+0000 7f387ecc8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T10:50:53.923 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:50:53.941+0000 7f387ecc8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:50:53.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:50:53.941+0000 7f387ecc8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:50:53.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:50:53.941+0000 7f387ecc8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:50:53.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:50:53.941+0000 7f387ecc8640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T10:50:56.769 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T10:50:56.802 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:50:56.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:50:56.912+0000 7f7e8b237640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T10:50:56.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:50:56.912+0000 7f7e8b237640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T10:50:56.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:50:56.912+0000 7f7e8b237640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:50:56.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:50:56.912+0000 7f7e8b237640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:50:56.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:50:56.912+0000 7f7e8b237640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:50:56.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:50:56.912+0000 7f7e8b237640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T10:50:59.742 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T10:50:59.772 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:50:59.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:50:59.880+0000 7f5e61208640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T10:50:59.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:50:59.881+0000 7f5e61208640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T10:50:59.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:50:59.881+0000 7f5e61208640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:50:59.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:50:59.881+0000 7f5e61208640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:50:59.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:50:59.881+0000 7f5e61208640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:50:59.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:50:59.881+0000 7f5e61208640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T10:51:02.712 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T10:51:02.742 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T10:51:02.742 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-03-29T10:51:02.742 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T10:51:02.775 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-03-29T10:51:02.775 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T10:51:02.846 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-03-29T10:51:02.846 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T10:51:02.914 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-03-29T10:51:02.914 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T10:51:02.983 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-03-29T10:51:02.983 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T10:51:03.050 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-29T10:51:03.050 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T10:51:03.080 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-29T10:51:03.080 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T10:51:03.149 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-29T10:51:03.150 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T10:51:03.219 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-29T10:51:03.219 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T10:51:03.287 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-29T10:51:03.287 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T10:51:03.356 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-03-29T10:51:03.357 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T10:51:03.375 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-29T10:51:03.375 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T10:51:03.412 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T10:51:03.412 DEBUG:teuthology.orchestra.run.smithi112:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:51:03.417 DEBUG:teuthology.orchestra.run.smithi118:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:51:03.456 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr.y] 2024-03-29T10:51:03.456 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQD5nAZmYieOORAAJUxzLNc8Nwrf0agOiVGIpw== 2024-03-29T10:51:03.456 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.0] 2024-03-29T10:51:03.456 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQD9nAZmvYc/FxAAC5zsqGsGnHvPL08VLiqoCA== 2024-03-29T10:51:03.456 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.1] 2024-03-29T10:51:03.456 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQAAnQZmTEDjFBAAE8PdCb6ug5KSB1BuyW3fSg== 2024-03-29T10:51:03.457 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.2] 2024-03-29T10:51:03.457 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQADnQZmVoknExAAypq+h+9F0GWrQ+OAYTItVA== 2024-03-29T10:51:03.457 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.3] 2024-03-29T10:51:03.457 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQAGnQZmmIqTEBAAn3JsWHD3PCHuMf06dou91A== 2024-03-29T10:51:03.457 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr.x] 2024-03-29T10:51:03.457 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQD6nAZmaPMRAxAA5B6Mk0/I8XPxD1YXkf0P9w== 2024-03-29T10:51:03.457 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.4] 2024-03-29T10:51:03.457 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQAKnQZmwAW6ORAA5Gqe64F/pWK873sg18AHxQ== 2024-03-29T10:51:03.457 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.5] 2024-03-29T10:51:03.457 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQANnQZmWhovOBAAPW8jTvSDXMe3obS16iGeUA== 2024-03-29T10:51:03.457 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.6] 2024-03-29T10:51:03.457 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQAQnQZmyLd1NhAAPkNb6PCg60jXuOI+Wwf07Q== 2024-03-29T10:51:03.457 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.7] 2024-03-29T10:51:03.457 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQATnQZmDM2GNBAA3dpdJFJojQ/CXVOdtdTmNQ== 2024-03-29T10:51:03.458 INFO:teuthology.orchestra.run.smithi112.stdout:[client.0] 2024-03-29T10:51:03.458 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQD6nAZmwyG8BRAAExFjbEOs8A6fAZQEODqM4A== 2024-03-29T10:51:03.458 INFO:teuthology.orchestra.run.smithi112.stdout:[client.1] 2024-03-29T10:51:03.458 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQD6nAZmcsYkCRAAFi18VjdBDhnaV5gedqdjyA== 2024-03-29T10:51:03.478 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr.y] 2024-03-29T10:51:03.479 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQD5nAZmYieOORAAJUxzLNc8Nwrf0agOiVGIpw== 2024-03-29T10:51:03.479 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.0] 2024-03-29T10:51:03.479 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQD9nAZmvYc/FxAAC5zsqGsGnHvPL08VLiqoCA== 2024-03-29T10:51:03.479 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.1] 2024-03-29T10:51:03.479 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAAnQZmTEDjFBAAE8PdCb6ug5KSB1BuyW3fSg== 2024-03-29T10:51:03.479 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.2] 2024-03-29T10:51:03.479 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQADnQZmVoknExAAypq+h+9F0GWrQ+OAYTItVA== 2024-03-29T10:51:03.479 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.3] 2024-03-29T10:51:03.479 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAGnQZmmIqTEBAAn3JsWHD3PCHuMf06dou91A== 2024-03-29T10:51:03.479 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr.x] 2024-03-29T10:51:03.479 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQD6nAZmaPMRAxAA5B6Mk0/I8XPxD1YXkf0P9w== 2024-03-29T10:51:03.479 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.4] 2024-03-29T10:51:03.479 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAKnQZmwAW6ORAA5Gqe64F/pWK873sg18AHxQ== 2024-03-29T10:51:03.480 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.5] 2024-03-29T10:51:03.480 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQANnQZmWhovOBAAPW8jTvSDXMe3obS16iGeUA== 2024-03-29T10:51:03.480 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.6] 2024-03-29T10:51:03.480 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAQnQZmyLd1NhAAPkNb6PCg60jXuOI+Wwf07Q== 2024-03-29T10:51:03.480 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.7] 2024-03-29T10:51:03.480 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQATnQZmDM2GNBAA3dpdJFJojQ/CXVOdtdTmNQ== 2024-03-29T10:51:03.480 INFO:teuthology.orchestra.run.smithi118.stdout:[client.0] 2024-03-29T10:51:03.480 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQD6nAZmwyG8BRAAExFjbEOs8A6fAZQEODqM4A== 2024-03-29T10:51:03.480 INFO:teuthology.orchestra.run.smithi118.stdout:[client.1] 2024-03-29T10:51:03.480 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQD6nAZmcsYkCRAAFi18VjdBDhnaV5gedqdjyA== 2024-03-29T10:51:03.481 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:51:03.502 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:51:03.575 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:03.577 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:03.629 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:03.631 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:03.682 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:03.684 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:03.736 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:03.738 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:03.789 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:51:03.791 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:51:03.843 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:03.846 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:03.898 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:03.900 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:03.951 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:03.954 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:04.005 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:04.007 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:04.059 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:51:04.061 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:51:04.116 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:51:04.119 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:51:04.173 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T10:51:04.173 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T10:51:04.200 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:51:04.296 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T10:51:04.323 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T10:51:04.391 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:51:04.488 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T10:51:04.514 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T10:51:04.542 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:51:04.639 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T10:51:04.665 DEBUG:teuthology.orchestra.run.smithi112:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:04.667 DEBUG:teuthology.orchestra.run.smithi118:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:04.720 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T10:51:04.720 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T10:51:04.721 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T10:51:04.723 INFO:tasks.ceph.mon.a:Started 2024-03-29T10:51:04.723 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T10:51:04.723 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T10:51:04.725 INFO:tasks.ceph.mon.c:Started 2024-03-29T10:51:04.725 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T10:51:04.725 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T10:51:04.764 INFO:tasks.ceph.mon.b:Started 2024-03-29T10:51:04.764 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T10:51:04.764 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T10:51:04.764 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T10:51:04.766 INFO:tasks.ceph.mgr.y:Started 2024-03-29T10:51:04.767 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T10:51:04.767 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T10:51:04.769 INFO:tasks.ceph.mgr.x:Started 2024-03-29T10:51:04.769 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T10:51:04.769 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T10:51:04.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:04.890+0000 7f028e9cf640 10 monclient: get_monmap_and_config 2024-03-29T10:51:04.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:04.890+0000 7f028e9cf640 10 monclient: build_initial_monmap 2024-03-29T10:51:04.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:04.890+0000 7f028e9cf640 10 monclient: monmap: 2024-03-29T10:51:04.872 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-03-29T10:51:04.872 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:04.872 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-03-29T10:51:04.891169+0000 2024-03-29T10:51:04.872 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-03-29T10:51:04.891169+0000 2024-03-29T10:51:04.872 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:04.872 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:04.872 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:04.872 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:51:04.872 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:51:04.872 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-03-29T10:51:04.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:04.891+0000 7f028e9cf640 1 Processor -- start 2024-03-29T10:51:04.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:04.891+0000 7f028e9cf640 1 -- start start 2024-03-29T10:51:04.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:04.891+0000 7f028e9cf640 10 monclient: init 2024-03-29T10:51:04.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:04.891+0000 7f028e9cf640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:04.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:04.891+0000 7f028e9cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:51:04.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:04.891+0000 7f028e9cf640 10 monclient(hunting): picked mon.noname-b con 0x7f02880b8240 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:04.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:04.891+0000 7f028e9cf640 10 monclient(hunting): picked mon.noname-c con 0x7f02880b1f60 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:04.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:04.891+0000 7f028e9cf640 10 monclient(hunting): picked mon.noname-a con 0x7f028809c8c0 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:04.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:04.891+0000 7f028e9cf640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f028809bd80 con 0x7f028809c8c0 2024-03-29T10:51:04.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:04.891+0000 7f028e9cf640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f028809b960 con 0x7f02880b1f60 2024-03-29T10:51:04.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:04.891+0000 7f028e9cf640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02880b2ec0 con 0x7f02880b8240 2024-03-29T10:51:04.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:04.891+0000 7f028e9cf640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:04.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:04.891+0000 7f028e9cf640 10 monclient(hunting): authenticate will time out at 820.195878s 2024-03-29T10:51:04.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:04.891+0000 7f028cf6c640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f028809c8c0 0x7f028814d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:42474/0 (socket says 172.21.15.112:42474) 2024-03-29T10:51:04.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:04.891+0000 7f0287fff640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f02880b8240 0x7f028813f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:40684/0 (socket says 172.21.15.112:40684) 2024-03-29T10:51:04.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:04.891+0000 7f028cf6c640 1 -- 172.21.15.112:0/27469992 learned_addr learned my addr 172.21.15.112:0/27469992 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:51:07.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:07.892+0000 7f0285ffb640 10 monclient(hunting): tick 2024-03-29T10:51:07.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:07.892+0000 7f0285ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-29T10:51:07.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:07.892+0000 7f0285ffb640 1 monclient(hunting): continuing hunt 2024-03-29T10:51:07.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:07.892+0000 7f0285ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T10:51:07.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:07.892+0000 7f0285ffb640 1 -- 172.21.15.112:0/27469992 >> v1:172.21.15.112:6790/0 conn(0x7f02880b8240 legacy=0x7f028813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:07.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:07.892+0000 7f0285ffb640 1 -- 172.21.15.112:0/27469992 >> v1:172.21.15.118:6789/0 conn(0x7f02880b1f60 legacy=0x7f028809c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:07.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:07.892+0000 7f0285ffb640 1 -- 172.21.15.112:0/27469992 >> v1:172.21.15.112:6789/0 conn(0x7f028809c8c0 legacy=0x7f028814d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:07.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:07.892+0000 7f0285ffb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:51:07.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:07.892+0000 7f0285ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f02740036e0 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:07.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:07.892+0000 7f0285ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f02740072b0 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:07.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:07.892+0000 7f0285ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f02880b8240 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:07.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:07.892+0000 7f0285ffb640 1 -- 172.21.15.112:0/27469992 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f028813fb60 con 0x7f02740072b0 2024-03-29T10:51:07.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:07.892+0000 7f0285ffb640 1 -- 172.21.15.112:0/27469992 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f027400f8b0 con 0x7f02740036e0 2024-03-29T10:51:07.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:07.892+0000 7f0285ffb640 1 -- 172.21.15.112:0/27469992 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0274010b10 con 0x7f02880b8240 2024-03-29T10:51:07.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:07.892+0000 7f0285ffb640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:10.034 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.050+0000 7f0286ffd640 1 -- 172.21.15.112:0/27469992 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2465835164 0 0) 0x7f028813fb60 con 0x7f02740072b0 2024-03-29T10:51:10.034 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.050+0000 7f0286ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:10.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.050+0000 7f0286ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:10.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.050+0000 7f0286ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:10.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.050+0000 7f0286ffd640 10 monclient(hunting): my global_id is 4101 2024-03-29T10:51:10.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.050+0000 7f0286ffd640 1 -- 172.21.15.112:0/27469992 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0270003520 con 0x7f02740072b0 2024-03-29T10:51:10.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 1 -- 172.21.15.112:0/27469992 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769529535 0 0) 0x7f027400f8b0 con 0x7f02740036e0 2024-03-29T10:51:10.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:10.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:10.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:10.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 10 monclient(hunting): my global_id is 4105 2024-03-29T10:51:10.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 1 -- 172.21.15.112:0/27469992 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f028813fb60 con 0x7f02740036e0 2024-03-29T10:51:10.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 1 -- 172.21.15.112:0/27469992 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 730249061 0 0) 0x7f028813fb60 con 0x7f02740036e0 2024-03-29T10:51:10.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 1 -- 172.21.15.112:0/27469992 >> v1:172.21.15.112:6790/0 conn(0x7f02880b8240 legacy=0x7f027400d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:10.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 1 -- 172.21.15.112:0/27469992 >> v1:172.21.15.112:6789/0 conn(0x7f02740072b0 legacy=0x7f0274009720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:10.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:10.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 1 monclient: found mon.noname-c 2024-03-29T10:51:10.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:10.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T10:51:10.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 1 -- 172.21.15.112:0/27469992 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0274011d70 con 0x7f02740036e0 2024-03-29T10:51:10.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 10 monclient: _finish_auth 0 2024-03-29T10:51:10.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:10.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:10.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 1 -- 172.21.15.112:0/27469992 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f0278001610 con 0x7f02740036e0 2024-03-29T10:51:10.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:10.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:10.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 10 monclient: dump: 2024-03-29T10:51:10.037 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-03-29T10:51:10.037 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:10.037 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.037 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.037 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:10.037 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:10.037 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:10.037 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:10.037 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:10.037 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-03-29T10:51:10.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f028e9cf640 5 monclient: authenticate success, global_id 4105 2024-03-29T10:51:10.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f028e9cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:10.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 1 -- 172.21.15.112:0/27469992 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0278001770 con 0x7f02740036e0 2024-03-29T10:51:10.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:10.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 1 -- 172.21.15.112:0/27469992 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f0278001b80 con 0x7f02740036e0 2024-03-29T10:51:10.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:10.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:10.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f0286ffd640 10 monclient: dump: 2024-03-29T10:51:10.038 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-03-29T10:51:10.038 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:10.038 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.038 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.038 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:10.038 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:10.038 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:10.039 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:10.039 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:10.039 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-03-29T10:51:10.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f028e9cf640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:10.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.051+0000 7f028e9cf640 4 set_mon_vals no callback set 2024-03-29T10:51:10.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.052+0000 7f028e9cf640 10 monclient: shutdown 2024-03-29T10:51:10.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.052+0000 7f028e9cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:10.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.052+0000 7f028e9cf640 1 -- 172.21.15.112:0/27469992 >> v1:172.21.15.118:6789/0 conn(0x7f02740036e0 legacy=0x7f0274005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:10.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.052+0000 7f028e9cf640 1 -- 172.21.15.112:0/27469992 shutdown_connections 2024-03-29T10:51:10.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.052+0000 7f028e9cf640 1 -- 172.21.15.112:0/27469992 >> 172.21.15.112:0/27469992 conn(0x7f0288066b00 msgr2=0x7f0288066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:10.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.052+0000 7f028e9cf640 1 -- 172.21.15.112:0/27469992 shutdown_connections 2024-03-29T10:51:10.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.052+0000 7f028e9cf640 1 -- 172.21.15.112:0/27469992 wait complete. 2024-03-29T10:51:10.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.053+0000 7f028e9cf640 10 monclient: build_initial_monmap 2024-03-29T10:51:10.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.053+0000 7f028e9cf640 10 monclient: monmap: 2024-03-29T10:51:10.044 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-03-29T10:51:10.045 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:10.045 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-03-29T10:51:10.045 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-03-29T10:51:10.045 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:10.045 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:10.045 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:10.045 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:51:10.045 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:51:10.045 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-03-29T10:51:10.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.053+0000 7f028e9cf640 1 librados: starting msgr at 2024-03-29T10:51:10.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.053+0000 7f028e9cf640 1 librados: starting objecter 2024-03-29T10:51:10.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.054+0000 7f028e9cf640 1 Processor -- start 2024-03-29T10:51:10.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.054+0000 7f028e9cf640 1 -- start start 2024-03-29T10:51:10.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.054+0000 7f028e9cf640 1 librados: setting wanted keys 2024-03-29T10:51:10.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.054+0000 7f028e9cf640 1 librados: calling monclient init 2024-03-29T10:51:10.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.054+0000 7f028e9cf640 10 monclient: init 2024-03-29T10:51:10.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f028e9cf640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:10.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f028e9cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:10.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f028e9cf640 10 monclient(hunting): picked mon.noname-c con 0x7f02740036e0 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:10.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f028e9cf640 10 monclient(hunting): picked mon.noname-a con 0x7f02740072b0 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:10.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f028e9cf640 10 monclient(hunting): picked mon.noname-b con 0x7f02880b8240 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:10.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f0287fff640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f02740036e0 0x7f02880b0a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59182/0 (socket says 172.21.15.112:59182) 2024-03-29T10:51:10.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f0287fff640 1 -- 172.21.15.112:0/2862565203 learned_addr learned my addr 172.21.15.112:0/2862565203 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:51:10.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f02877fe640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f02740072b0 0x7f02881d7a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:40696/0 (socket says 172.21.15.112:40696) 2024-03-29T10:51:10.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f028e9cf640 1 -- 172.21.15.112:0/2862565203 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02880b1190 con 0x7f02740072b0 2024-03-29T10:51:10.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f028e9cf640 1 -- 172.21.15.112:0/2862565203 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02881d75e0 con 0x7f02880b8240 2024-03-29T10:51:10.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f028e9cf640 1 -- 172.21.15.112:0/2862565203 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02880ad2a0 con 0x7f02740036e0 2024-03-29T10:51:10.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f0284ff9640 10 client.?.objecter ms_handle_connect 0x7f02740072b0 2024-03-29T10:51:10.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f0284ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:10.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f0284ff9640 10 client.?.objecter ms_handle_connect 0x7f02740036e0 2024-03-29T10:51:10.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f0284ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:10.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f0284ff9640 10 client.?.objecter ms_handle_connect 0x7f02880b8240 2024-03-29T10:51:10.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f0284ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:10.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f028e9cf640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:10.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f028e9cf640 10 monclient(hunting): authenticate will time out at 825.359807s 2024-03-29T10:51:10.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f0284ff9640 1 -- 172.21.15.112:0/2862565203 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076012809 0 0) 0x7f02881d75e0 con 0x7f02880b8240 2024-03-29T10:51:10.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f0284ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:10.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f0284ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:10.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f0284ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:10.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f0284ff9640 10 monclient(hunting): my global_id is 4111 2024-03-29T10:51:10.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f0284ff9640 1 -- 172.21.15.112:0/2862565203 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0274011460 con 0x7f02880b8240 2024-03-29T10:51:10.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f0284ff9640 1 -- 172.21.15.112:0/2862565203 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201720194 0 0) 0x7f02880ad2a0 con 0x7f02740036e0 2024-03-29T10:51:10.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f0284ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:10.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f0284ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:10.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f0284ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:10.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f0284ff9640 10 monclient(hunting): my global_id is 4106 2024-03-29T10:51:10.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.055+0000 7f0284ff9640 1 -- 172.21.15.112:0/2862565203 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02881d75e0 con 0x7f02740036e0 2024-03-29T10:51:10.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 1 -- 172.21.15.112:0/2862565203 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2022578861 0 0) 0x7f0274011460 con 0x7f02880b8240 2024-03-29T10:51:10.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 1 -- 172.21.15.112:0/2862565203 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02880ad2a0 con 0x7f02880b8240 2024-03-29T10:51:10.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 1 -- 172.21.15.112:0/2862565203 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f0280001aa0 con 0x7f02880b8240 2024-03-29T10:51:10.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:10.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:10.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 10 monclient(hunting): dump: 2024-03-29T10:51:10.048 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-03-29T10:51:10.048 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:10.048 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.048 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.049 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:10.049 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:10.049 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:10.049 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:10.049 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:10.049 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-03-29T10:51:10.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:10.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 1 -- 172.21.15.112:0/2862565203 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3377550583 0 0) 0x7f02881d75e0 con 0x7f02740036e0 2024-03-29T10:51:10.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 1 -- 172.21.15.112:0/2862565203 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0274011460 con 0x7f02740036e0 2024-03-29T10:51:10.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 1 -- 172.21.15.112:0/2862565203 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f0278001ac0 con 0x7f02740036e0 2024-03-29T10:51:10.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:10.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:10.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 10 monclient(hunting): dump: 2024-03-29T10:51:10.049 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-03-29T10:51:10.049 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:10.050 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.050 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.050 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:10.050 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:10.050 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:10.050 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:10.050 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:10.050 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-03-29T10:51:10.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 1 -- 172.21.15.112:0/2862565203 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 626586925 0 0) 0x7f02880ad2a0 con 0x7f02880b8240 2024-03-29T10:51:10.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 1 -- 172.21.15.112:0/2862565203 >> v1:172.21.15.112:6790/0 conn(0x7f02740036e0 legacy=0x7f02880b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:10.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 1 -- 172.21.15.112:0/2862565203 >> v1:172.21.15.112:6789/0 conn(0x7f02740072b0 legacy=0x7f02881d7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:10.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:51:10.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 1 monclient: found mon.b 2024-03-29T10:51:10.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:10.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:10.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 1 -- 172.21.15.112:0/2862565203 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0288139f20 con 0x7f02880b8240 2024-03-29T10:51:10.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 10 monclient: _finish_auth 0 2024-03-29T10:51:10.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:51:10.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:10.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f028e9cf640 5 monclient: authenticate success, global_id 4111 2024-03-29T10:51:10.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f028e9cf640 10 monclient: _renew_subs 2024-03-29T10:51:10.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f028e9cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:10.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f028e9cf640 1 -- 172.21.15.112:0/2862565203 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02880b01b0 con 0x7f02880b8240 2024-03-29T10:51:10.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f028e9cf640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:10.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f028e9cf640 10 monclient: _renew_subs 2024-03-29T10:51:10.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f028e9cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:10.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f028e9cf640 1 -- 172.21.15.112:0/2862565203 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02881d8360 con 0x7f02880b8240 2024-03-29T10:51:10.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f028e9cf640 1 librados: init done 2024-03-29T10:51:10.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 1 -- 172.21.15.112:0/2862565203 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0280002030 con 0x7f02880b8240 2024-03-29T10:51:10.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:10.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 1 -- 172.21.15.112:0/2862565203 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f0280004d00 con 0x7f02880b8240 2024-03-29T10:51:10.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:10.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:10.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 10 monclient: dump: 2024-03-29T10:51:10.052 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-03-29T10:51:10.052 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:10.052 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.052 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.052 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:10.052 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:10.052 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:10.052 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:10.052 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:10.052 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-03-29T10:51:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f02867fc640 4 set_mon_vals no callback set 2024-03-29T10:51:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 1 -- 172.21.15.112:0/2862565203 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f02800051f0 con 0x7f02880b8240 2024-03-29T10:51:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 1 -- 172.21.15.112:0/2862565203 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1975307637 0 0) 0x7f0280006720 con 0x7f02880b8240 2024-03-29T10:51:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 10 client.4111.objecter ms_dispatch 0x7f0288000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T10:51:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 3 client.4111.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T10:51:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 3 client.4111.objecter handle_osd_map decoding full epoch 1 2024-03-29T10:51:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.056+0000 7f0284ff9640 20 client.4111.objecter dump_active .. 0 homeless 2024-03-29T10:51:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.058+0000 7f028e9cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.058+0000 7f028e9cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.058+0000 7f028e9cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.058+0000 7f028e9cf640 1 -- 172.21.15.112:0/2862565203 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0268005080 con 0x7f02880b8240 2024-03-29T10:51:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.060+0000 7f0284ff9640 1 -- 172.21.15.112:0/2862565203 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0280006a00 con 0x7f02880b8240 2024-03-29T10:51:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.060+0000 7f0284ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.060+0000 7f0284ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:10.112 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.130+0000 7f028e9cf640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:51:10.113 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.131+0000 7f028e9cf640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:51:10.113 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.131+0000 7f028e9cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:10.113 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.131+0000 7f028e9cf640 1 -- 172.21.15.112:0/2862565203 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f0268005330 con 0x7f02880b8240 2024-03-29T10:51:10.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.134+0000 7f0284ff9640 1 -- 172.21.15.112:0/2862565203 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f0280005710 con 0x7f02880b8240 2024-03-29T10:51:10.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.134+0000 7f0284ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:51:10.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.134+0000 7f0284ff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T10:51:10.115 INFO:teuthology.orchestra.run.smithi112.stderr:adjusted tunables profile to default 2024-03-29T10:51:10.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.134+0000 7f028e9cf640 10 librados: watch_flush enter 2024-03-29T10:51:10.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.135+0000 7f028e9cf640 10 librados: watch_flush exit 2024-03-29T10:51:10.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.135+0000 7f028e9cf640 20 client.4111.objecter shutdown clearing up homeless session... 2024-03-29T10:51:10.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.135+0000 7f028e9cf640 10 client.4111.objecter successfully canceled tick 2024-03-29T10:51:10.118 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.135+0000 7f028e9cf640 10 monclient: shutdown 2024-03-29T10:51:10.118 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.135+0000 7f028e9cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:10.118 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.135+0000 7f028e9cf640 1 -- 172.21.15.112:0/2862565203 >> v1:172.21.15.118:6789/0 conn(0x7f02880b8240 legacy=0x7f02881391d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:10.118 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.135+0000 7f028e9cf640 1 -- 172.21.15.112:0/2862565203 shutdown_connections 2024-03-29T10:51:10.118 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.135+0000 7f028e9cf640 1 -- 172.21.15.112:0/2862565203 >> 172.21.15.112:0/2862565203 conn(0x7f0288066b00 msgr2=0x7f02880b4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:10.118 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.135+0000 7f028e9cf640 1 -- 172.21.15.112:0/2862565203 shutdown_connections 2024-03-29T10:51:10.118 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.135+0000 7f028e9cf640 1 -- 172.21.15.112:0/2862565203 wait complete. 2024-03-29T10:51:10.118 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:10.135+0000 7f028e9cf640 1 librados: shutdown 2024-03-29T10:51:10.128 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T10:51:10.128 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T10:51:10.128 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-03-29T10:51:10.128 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T10:51:10.155 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-03-29T10:51:10.156 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T10:51:10.222 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-03-29T10:51:10.222 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T10:51:10.290 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-03-29T10:51:10.290 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T10:51:10.357 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-29T10:51:10.357 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T10:51:10.384 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-29T10:51:10.385 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T10:51:10.452 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-29T10:51:10.452 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T10:51:10.518 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-29T10:51:10.518 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T10:51:10.526 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:10.544+0000 7f0c627b8040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:51:10.580 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:10.599+0000 7f0c627b8040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:51:10.586 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new 84afddc6-02ba-4dda-8b9a-ea9b91510482 0 2024-03-29T10:51:10.595 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:10.614+0000 7f03b67b6040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:51:10.652 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:10.671+0000 7f03b67b6040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:51:10.694 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.712+0000 7f4135233640 10 monclient: get_monmap_and_config 2024-03-29T10:51:10.694 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.713+0000 7f4135233640 10 monclient: build_initial_monmap 2024-03-29T10:51:10.694 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.713+0000 7f4135233640 10 monclient: monmap: 2024-03-29T10:51:10.694 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2024-03-29T10:51:10.694 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:10.694 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:51:10.713338+0000 2024-03-29T10:51:10.694 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:51:10.713338+0000 2024-03-29T10:51:10.695 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:10.695 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:10.695 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:10.695 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:51:10.695 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:51:10.695 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:10.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.713+0000 7f4135233640 1 Processor -- start 2024-03-29T10:51:10.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.713+0000 7f4135233640 1 -- start start 2024-03-29T10:51:10.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.713+0000 7f4135233640 10 monclient: init 2024-03-29T10:51:10.704 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f4135233640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:10.704 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f4135233640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:51:10.704 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f4135233640 10 monclient(hunting): picked mon.noname-c con 0x7f413009ac40 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:10.704 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f4135233640 10 monclient(hunting): picked mon.noname-a con 0x7f413009c870 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:10.704 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f4135233640 10 monclient(hunting): picked mon.noname-b con 0x7f41300b4360 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:10.704 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f4135233640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413009dec0 con 0x7f413009c870 2024-03-29T10:51:10.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f4135233640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413009da30 con 0x7f413009ac40 2024-03-29T10:51:10.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f4135233640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41300b39c0 con 0x7f41300b4360 2024-03-29T10:51:10.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f4135233640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:10.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f4135233640 10 monclient(hunting): authenticate will time out at 811.032472s 2024-03-29T10:51:10.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412ed9d640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f413009ac40 0x7f41300b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:44602/0 (socket says 172.21.15.118:44602) 2024-03-29T10:51:10.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412ed9d640 1 -- 172.21.15.118:0/3543840684 learned_addr learned my addr 172.21.15.118:0/3543840684 (peer_addr_for_me v1:172.21.15.118:0/0) 2024-03-29T10:51:10.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 1 -- 172.21.15.118:0/3543840684 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122122720 0 0) 0x7f413009da30 con 0x7f413009ac40 2024-03-29T10:51:10.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:51:10.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:10.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:10.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 10 monclient(hunting): my global_id is 4114 2024-03-29T10:51:10.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 1 -- 172.21.15.118:0/3543840684 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4118003520 con 0x7f413009ac40 2024-03-29T10:51:10.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 1 -- 172.21.15.118:0/3543840684 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816979760 0 0) 0x7f41300b39c0 con 0x7f41300b4360 2024-03-29T10:51:10.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:51:10.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:10.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:10.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 10 monclient(hunting): my global_id is 4109 2024-03-29T10:51:10.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 1 -- 172.21.15.118:0/3543840684 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f413009da30 con 0x7f41300b4360 2024-03-29T10:51:10.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 1 -- 172.21.15.118:0/3543840684 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 870335519 0 0) 0x7f4120001110 con 0x7f413009c870 2024-03-29T10:51:10.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:51:10.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:10.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:10.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 10 monclient(hunting): my global_id is 4107 2024-03-29T10:51:10.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 1 -- 172.21.15.118:0/3543840684 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41300b39c0 con 0x7f413009c870 2024-03-29T10:51:10.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 1 -- 172.21.15.118:0/3543840684 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3733931407 0 0) 0x7f4118003520 con 0x7f413009ac40 2024-03-29T10:51:10.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 1 -- 172.21.15.118:0/3543840684 >> v1:172.21.15.112:6790/0 conn(0x7f41300b4360 legacy=0x7f413014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:10.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 1 -- 172.21.15.118:0/3543840684 >> v1:172.21.15.112:6789/0 conn(0x7f413009c870 legacy=0x7f413013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:10.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 10 monclient: _finish_hunting 0 2024-03-29T10:51:10.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 1 monclient: found mon.noname-c 2024-03-29T10:51:10.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:10.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T10:51:10.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 1 -- 172.21.15.118:0/3543840684 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41300b5160 con 0x7f413009ac40 2024-03-29T10:51:10.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 10 monclient: _finish_auth 0 2024-03-29T10:51:10.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 10 monclient: _check_auth_tickets 2024-03-29T10:51:10.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:10.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 1 -- 172.21.15.118:0/3543840684 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f4124001690 con 0x7f413009ac40 2024-03-29T10:51:10.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f4135233640 5 monclient: authenticate success, global_id 4114 2024-03-29T10:51:10.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f4135233640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:10.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:10.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:10.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f412dd9b640 10 monclient: dump: 2024-03-29T10:51:10.708 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:10.708 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:10.708 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.708 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.708 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:10.708 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:10.708 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:10.708 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:10.708 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:10.708 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:10.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.714+0000 7f4135233640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:10.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.715+0000 7f412dd9b640 1 -- 172.21.15.118:0/3543840684 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4124001870 con 0x7f413009ac40 2024-03-29T10:51:10.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.715+0000 7f412dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:10.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.715+0000 7f412dd9b640 1 -- 172.21.15.118:0/3543840684 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f4124001c80 con 0x7f413009ac40 2024-03-29T10:51:10.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.715+0000 7f4135233640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:10.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.715+0000 7f4135233640 4 set_mon_vals no callback set 2024-03-29T10:51:10.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.715+0000 7f412dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:10.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.715+0000 7f412dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:10.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.715+0000 7f412dd9b640 10 monclient: dump: 2024-03-29T10:51:10.709 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:10.709 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:10.709 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.709 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.709 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:10.709 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:10.709 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:10.709 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:10.710 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:10.710 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:10.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.715+0000 7f4135233640 10 monclient: shutdown 2024-03-29T10:51:10.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.715+0000 7f4135233640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:10.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.715+0000 7f4135233640 1 -- 172.21.15.118:0/3543840684 >> v1:172.21.15.118:6789/0 conn(0x7f413009ac40 legacy=0x7f41300b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:10.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.715+0000 7f4135233640 1 -- 172.21.15.118:0/3543840684 shutdown_connections 2024-03-29T10:51:10.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.715+0000 7f4135233640 1 -- 172.21.15.118:0/3543840684 >> 172.21.15.118:0/3543840684 conn(0x7f4130066b00 msgr2=0x7f4130066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:10.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.715+0000 7f4135233640 1 -- 172.21.15.118:0/3543840684 shutdown_connections 2024-03-29T10:51:10.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.715+0000 7f4135233640 1 -- 172.21.15.118:0/3543840684 wait complete. 2024-03-29T10:51:10.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.716+0000 7f4135233640 10 monclient: build_initial_monmap 2024-03-29T10:51:10.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.716+0000 7f4135233640 10 monclient: monmap: 2024-03-29T10:51:10.710 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2024-03-29T10:51:10.710 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:10.711 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 0.000000 2024-03-29T10:51:10.711 INFO:teuthology.orchestra.run.smithi118.stderr:created 0.000000 2024-03-29T10:51:10.711 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:10.711 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:10.711 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:10.711 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:51:10.711 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:51:10.711 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:10.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.716+0000 7f4135233640 1 librados: starting msgr at 2024-03-29T10:51:10.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.716+0000 7f4135233640 1 librados: starting objecter 2024-03-29T10:51:10.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.716+0000 7f4135233640 1 Processor -- start 2024-03-29T10:51:10.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.716+0000 7f4135233640 1 -- start start 2024-03-29T10:51:10.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.716+0000 7f4135233640 1 librados: setting wanted keys 2024-03-29T10:51:10.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.716+0000 7f4135233640 1 librados: calling monclient init 2024-03-29T10:51:10.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.716+0000 7f4135233640 10 monclient: init 2024-03-29T10:51:10.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f4135233640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:10.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f4135233640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:10.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f4135233640 10 monclient(hunting): picked mon.noname-c con 0x7f413009ac40 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:10.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f4135233640 10 monclient(hunting): picked mon.noname-a con 0x7f41300adbe0 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:10.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f4135233640 10 monclient(hunting): picked mon.noname-b con 0x7f4130136a40 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:10.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f4135233640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41300ae720 con 0x7f41300adbe0 2024-03-29T10:51:10.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f4135233640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41300af7a0 con 0x7f4130136a40 2024-03-29T10:51:10.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f4135233640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41300ae200 con 0x7f413009ac40 2024-03-29T10:51:10.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f4135233640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:10.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f4135233640 10 monclient(hunting): authenticate will time out at 811.035655s 2024-03-29T10:51:10.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f412ed9d640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f413009ac40 0x7f41300ad3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.118:51738/0 (socket says 172.21.15.118:51738) 2024-03-29T10:51:10.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f412ed9d640 1 -- 172.21.15.118:0/2822140343 learned_addr learned my addr 172.21.15.118:0/2822140343 (peer_addr_for_me v1:172.21.15.118:0/0) 2024-03-29T10:51:10.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f412e59c640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f41300adbe0 0x7f4130136330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.118:36692/0 (socket says 172.21.15.118:36692) 2024-03-29T10:51:10.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f412f59e640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f4130136a40 0x7f41300b6510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:44618/0 (socket says 172.21.15.118:44618) 2024-03-29T10:51:10.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f41177fe640 10 client.?.objecter ms_handle_connect 0x7f4130136a40 2024-03-29T10:51:10.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f41177fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:10.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f41177fe640 10 client.?.objecter ms_handle_connect 0x7f413009ac40 2024-03-29T10:51:10.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f41177fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:10.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f41177fe640 10 client.?.objecter ms_handle_connect 0x7f41300adbe0 2024-03-29T10:51:10.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f41177fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:10.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107108882 0 0) 0x7f41300af7a0 con 0x7f4130136a40 2024-03-29T10:51:10.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f41177fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:10.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f41177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:10.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f41177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:10.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f41177fe640 10 monclient(hunting): my global_id is 4117 2024-03-29T10:51:10.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f411c004eb0 con 0x7f4130136a40 2024-03-29T10:51:10.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 382494442 0 0) 0x7f41300ae720 con 0x7f41300adbe0 2024-03-29T10:51:10.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f41177fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:10.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f41177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:10.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f41177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:10.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f41177fe640 10 monclient(hunting): my global_id is 4110 2024-03-29T10:51:10.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41300af7a0 con 0x7f41300adbe0 2024-03-29T10:51:10.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042325514 0 0) 0x7f41300ae200 con 0x7f413009ac40 2024-03-29T10:51:10.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f41177fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:10.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f41177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:10.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f41177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:10.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f41177fe640 10 monclient(hunting): my global_id is 4112 2024-03-29T10:51:10.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.717+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41300ae720 con 0x7f413009ac40 2024-03-29T10:51:10.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1117510146 0 0) 0x7f411c004eb0 con 0x7f4130136a40 2024-03-29T10:51:10.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41300ae200 con 0x7f4130136a40 2024-03-29T10:51:10.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f4128001b50 con 0x7f4130136a40 2024-03-29T10:51:10.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:10.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:10.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 10 monclient(hunting): dump: 2024-03-29T10:51:10.715 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:10.715 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:10.715 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.715 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.715 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:10.715 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:10.715 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:10.715 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:10.715 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:10.715 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:10.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:10.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 14638337 0 0) 0x7f41300af7a0 con 0x7f41300adbe0 2024-03-29T10:51:10.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f411c004eb0 con 0x7f41300adbe0 2024-03-29T10:51:10.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 606331646 0 0) 0x7f41300ae720 con 0x7f413009ac40 2024-03-29T10:51:10.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41300af7a0 con 0x7f413009ac40 2024-03-29T10:51:10.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f41200018e0 con 0x7f41300adbe0 2024-03-29T10:51:10.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:10.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:10.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 10 monclient(hunting): dump: 2024-03-29T10:51:10.716 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:10.716 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:10.716 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.716 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.716 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:10.716 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:10.716 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:10.716 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:10.716 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:10.716 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:10.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f4124001b80 con 0x7f413009ac40 2024-03-29T10:51:10.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:10.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:10.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 10 monclient(hunting): dump: 2024-03-29T10:51:10.717 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:10.717 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:10.717 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.717 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.717 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:10.717 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:10.717 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:10.717 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:10.717 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:10.717 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:10.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2088389283 0 0) 0x7f41300ae200 con 0x7f4130136a40 2024-03-29T10:51:10.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 >> v1:172.21.15.112:6790/0 conn(0x7f413009ac40 legacy=0x7f41300ad3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:10.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 >> v1:172.21.15.112:6789/0 conn(0x7f41300adbe0 legacy=0x7f4130136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:10.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 10 monclient: _finish_hunting 0 2024-03-29T10:51:10.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 1 monclient: found mon.b 2024-03-29T10:51:10.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:10.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:10.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41300b6c20 con 0x7f4130136a40 2024-03-29T10:51:10.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 10 monclient: _finish_auth 0 2024-03-29T10:51:10.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 10 monclient: _check_auth_tickets 2024-03-29T10:51:10.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:10.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f4135233640 5 monclient: authenticate success, global_id 4117 2024-03-29T10:51:10.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f4135233640 10 monclient: _renew_subs 2024-03-29T10:51:10.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f4135233640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:10.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f4135233640 1 -- 172.21.15.118:0/2822140343 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41300b6ef0 con 0x7f4130136a40 2024-03-29T10:51:10.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f4135233640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:10.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f4135233640 10 monclient: _renew_subs 2024-03-29T10:51:10.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f4135233640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:10.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f4135233640 1 -- 172.21.15.118:0/2822140343 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41301e3d10 con 0x7f4130136a40 2024-03-29T10:51:10.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f4135233640 1 librados: init done 2024-03-29T10:51:10.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41280020c0 con 0x7f4130136a40 2024-03-29T10:51:10.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:10.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f4128003b30 con 0x7f4130136a40 2024-03-29T10:51:10.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:10.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:10.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f412d59a640 4 set_mon_vals no callback set 2024-03-29T10:51:10.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 10 monclient: dump: 2024-03-29T10:51:10.719 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:10.719 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:10.719 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.719 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.720 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:10.720 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:10.720 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:10.720 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:10.720 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:10.720 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:10.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4128004020 con 0x7f4130136a40 2024-03-29T10:51:10.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 337152739 0 0) 0x7f41280067d0 con 0x7f4130136a40 2024-03-29T10:51:10.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 10 client.4117.objecter ms_dispatch 0x7f4130000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T10:51:10.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 3 client.4117.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T10:51:10.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 3 client.4117.objecter handle_osd_map decoding full epoch 2 2024-03-29T10:51:10.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.718+0000 7f41177fe640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-29T10:51:10.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.719+0000 7f4135233640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:10.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.719+0000 7f4135233640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:10.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.719+0000 7f4135233640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:10.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.719+0000 7f4135233640 1 -- 172.21.15.118:0/2822140343 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4110005080 con 0x7f4130136a40 2024-03-29T10:51:10.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.722+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4128004570 con 0x7f4130136a40 2024-03-29T10:51:10.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.722+0000 7f41177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:10.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.722+0000 7f41177fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:10.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.793+0000 7f4135233640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "84afddc6-02ba-4dda-8b9a-ea9b91510482", "id": 0}] 2024-03-29T10:51:10.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.793+0000 7f4135233640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "84afddc6-02ba-4dda-8b9a-ea9b91510482", "id": 0}] 2024-03-29T10:51:10.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.793+0000 7f4135233640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:10.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.793+0000 7f4135233640 1 -- 172.21.15.118:0/2822140343 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd new", "uuid": "84afddc6-02ba-4dda-8b9a-ea9b91510482", "id": 0} v 0) v1 -- 0x7f41100058b0 con 0x7f4130136a40 2024-03-29T10:51:10.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.796+0000 7f41177fe640 1 -- 172.21.15.118:0/2822140343 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "84afddc6-02ba-4dda-8b9a-ea9b91510482", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 2949108804 0 732707515) 0x7f4128006a60 con 0x7f4130136a40 2024-03-29T10:51:10.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.796+0000 7f41177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "84afddc6-02ba-4dda-8b9a-ea9b91510482", "id": 0}] 2024-03-29T10:51:10.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.796+0000 7f41177fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:10.778 INFO:teuthology.orchestra.run.smithi118.stdout:0 2024-03-29T10:51:10.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.797+0000 7f4135233640 10 librados: watch_flush enter 2024-03-29T10:51:10.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.797+0000 7f4135233640 10 librados: watch_flush exit 2024-03-29T10:51:10.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.797+0000 7f4135233640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-29T10:51:10.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.797+0000 7f4135233640 10 client.4117.objecter successfully canceled tick 2024-03-29T10:51:10.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.797+0000 7f4135233640 10 monclient: shutdown 2024-03-29T10:51:10.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.797+0000 7f4135233640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:10.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.797+0000 7f4135233640 1 -- 172.21.15.118:0/2822140343 >> v1:172.21.15.118:6789/0 conn(0x7f4130136a40 legacy=0x7f41300b6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:10.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.797+0000 7f4135233640 1 -- 172.21.15.118:0/2822140343 shutdown_connections 2024-03-29T10:51:10.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.797+0000 7f4135233640 1 -- 172.21.15.118:0/2822140343 >> 172.21.15.118:0/2822140343 conn(0x7f4130066b00 msgr2=0x7f413013d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:10.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.797+0000 7f4135233640 1 -- 172.21.15.118:0/2822140343 shutdown_connections 2024-03-29T10:51:10.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.797+0000 7f4135233640 1 -- 172.21.15.118:0/2822140343 wait complete. 2024-03-29T10:51:10.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.797+0000 7f4135233640 1 librados: shutdown 2024-03-29T10:51:10.789 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new 600a2c50-5341-4503-847b-fd2a2922a007 1 2024-03-29T10:51:10.886 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.906+0000 7fb38eec1640 10 monclient: get_monmap_and_config 2024-03-29T10:51:10.887 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.906+0000 7fb38eec1640 10 monclient: build_initial_monmap 2024-03-29T10:51:10.887 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.906+0000 7fb38eec1640 10 monclient: monmap: 2024-03-29T10:51:10.887 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2024-03-29T10:51:10.887 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:10.887 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:51:10.906404+0000 2024-03-29T10:51:10.887 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:51:10.906404+0000 2024-03-29T10:51:10.887 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:10.887 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:10.887 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:10.887 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:51:10.887 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:51:10.887 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:10.898 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.906+0000 7fb38eec1640 1 Processor -- start 2024-03-29T10:51:10.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.906+0000 7fb38eec1640 1 -- start start 2024-03-29T10:51:10.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.906+0000 7fb38eec1640 10 monclient: init 2024-03-29T10:51:10.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb38eec1640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:10.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb38eec1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:51:10.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb38eec1640 10 monclient(hunting): picked mon.noname-b con 0x7fb38809ac40 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:10.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb38eec1640 10 monclient(hunting): picked mon.noname-c con 0x7fb38809c870 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:10.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb38eec1640 10 monclient(hunting): picked mon.noname-a con 0x7fb3880b4360 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:10.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb38eec1640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb38809dec0 con 0x7fb3880b4360 2024-03-29T10:51:10.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb38eec1640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb38809da30 con 0x7fb38809c870 2024-03-29T10:51:10.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb38eec1640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3880b39c0 con 0x7fb38809ac40 2024-03-29T10:51:10.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb38eec1640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:10.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb38eec1640 10 monclient(hunting): authenticate will time out at 811.225689s 2024-03-29T10:51:10.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb387fff640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fb38809c870 0x7fb38813f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:44632/0 (socket says 172.21.15.118:44632) 2024-03-29T10:51:10.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb387fff640 1 -- 172.21.15.118:0/1214551507 learned_addr learned my addr 172.21.15.118:0/1214551507 (peer_addr_for_me v1:172.21.15.118:0/0) 2024-03-29T10:51:10.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb3877fe640 1 -- 172.21.15.118:0/1214551507 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023143783 0 0) 0x7fb38809da30 con 0x7fb38809c870 2024-03-29T10:51:10.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb3877fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:10.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb3877fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:10.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb3877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:10.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb3877fe640 10 monclient(hunting): my global_id is 4120 2024-03-29T10:51:10.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb3877fe640 1 -- 172.21.15.118:0/1214551507 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb378003520 con 0x7fb38809c870 2024-03-29T10:51:10.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb3877fe640 1 -- 172.21.15.118:0/1214551507 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348290027 0 0) 0x7fb380001490 con 0x7fb3880b4360 2024-03-29T10:51:10.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb3877fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:10.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb3877fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:10.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb3877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:10.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb3877fe640 10 monclient(hunting): my global_id is 4113 2024-03-29T10:51:10.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb3877fe640 1 -- 172.21.15.118:0/1214551507 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb38809da30 con 0x7fb3880b4360 2024-03-29T10:51:10.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.907+0000 7fb3877fe640 1 -- 172.21.15.118:0/1214551507 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129239892 0 0) 0x7fb3880b39c0 con 0x7fb38809ac40 2024-03-29T10:51:10.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:10.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:10.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:10.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 10 monclient(hunting): my global_id is 4115 2024-03-29T10:51:10.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 1 -- 172.21.15.118:0/1214551507 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb380001490 con 0x7fb38809ac40 2024-03-29T10:51:10.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 1 -- 172.21.15.118:0/1214551507 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2731022847 0 0) 0x7fb378003520 con 0x7fb38809c870 2024-03-29T10:51:10.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 1 -- 172.21.15.118:0/1214551507 >> v1:172.21.15.112:6790/0 conn(0x7fb38809ac40 legacy=0x7fb3880b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:10.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 1 -- 172.21.15.118:0/1214551507 >> v1:172.21.15.112:6789/0 conn(0x7fb3880b4360 legacy=0x7fb38814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:10.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 10 monclient: _finish_hunting 0 2024-03-29T10:51:10.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 1 monclient: found mon.noname-c 2024-03-29T10:51:10.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:10.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T10:51:10.902 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 1 -- 172.21.15.118:0/1214551507 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3880b5160 con 0x7fb38809c870 2024-03-29T10:51:10.902 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 10 monclient: _finish_auth 0 2024-03-29T10:51:10.902 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 10 monclient: _check_auth_tickets 2024-03-29T10:51:10.902 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:10.902 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb38eec1640 5 monclient: authenticate success, global_id 4120 2024-03-29T10:51:10.902 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 1 -- 172.21.15.118:0/1214551507 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb374001690 con 0x7fb38809c870 2024-03-29T10:51:10.902 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb38eec1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:10.902 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:10.902 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:10.902 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 10 monclient: dump: 2024-03-29T10:51:10.902 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:10.902 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:10.902 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.903 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.903 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:10.903 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:10.903 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:10.903 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:10.903 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:10.903 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:10.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb38eec1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:10.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 1 -- 172.21.15.118:0/1214551507 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb374001870 con 0x7fb38809c870 2024-03-29T10:51:10.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:10.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb38eec1640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:10.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 1 -- 172.21.15.118:0/1214551507 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb374001c80 con 0x7fb38809c870 2024-03-29T10:51:10.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:10.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb38eec1640 4 set_mon_vals no callback set 2024-03-29T10:51:10.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:10.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb3877fe640 10 monclient: dump: 2024-03-29T10:51:10.904 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:10.904 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:10.904 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.904 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.904 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:10.904 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:10.904 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:10.904 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:10.904 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:10.904 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:10.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb38eec1640 10 monclient: shutdown 2024-03-29T10:51:10.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb38eec1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:10.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb38eec1640 1 -- 172.21.15.118:0/1214551507 >> v1:172.21.15.118:6789/0 conn(0x7fb38809c870 legacy=0x7fb38813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:10.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb38eec1640 1 -- 172.21.15.118:0/1214551507 shutdown_connections 2024-03-29T10:51:10.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb38eec1640 1 -- 172.21.15.118:0/1214551507 >> 172.21.15.118:0/1214551507 conn(0x7fb388066b00 msgr2=0x7fb388066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:10.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb38eec1640 1 -- 172.21.15.118:0/1214551507 shutdown_connections 2024-03-29T10:51:10.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.908+0000 7fb38eec1640 1 -- 172.21.15.118:0/1214551507 wait complete. 2024-03-29T10:51:10.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.909+0000 7fb38eec1640 10 monclient: build_initial_monmap 2024-03-29T10:51:10.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.909+0000 7fb38eec1640 10 monclient: monmap: 2024-03-29T10:51:10.905 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2024-03-29T10:51:10.905 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:10.905 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 0.000000 2024-03-29T10:51:10.905 INFO:teuthology.orchestra.run.smithi118.stderr:created 0.000000 2024-03-29T10:51:10.905 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:10.905 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:10.905 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:10.906 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:51:10.906 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:51:10.906 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:10.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.909+0000 7fb38eec1640 1 librados: starting msgr at 2024-03-29T10:51:10.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.909+0000 7fb38eec1640 1 librados: starting objecter 2024-03-29T10:51:10.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.909+0000 7fb38eec1640 1 Processor -- start 2024-03-29T10:51:10.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.909+0000 7fb38eec1640 1 -- start start 2024-03-29T10:51:10.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.909+0000 7fb38eec1640 1 librados: setting wanted keys 2024-03-29T10:51:10.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.909+0000 7fb38eec1640 1 librados: calling monclient init 2024-03-29T10:51:10.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.909+0000 7fb38eec1640 10 monclient: init 2024-03-29T10:51:10.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb38eec1640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:10.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb38eec1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:51:10.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb38eec1640 10 monclient(hunting): picked mon.noname-b con 0x7fb38809ac40 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:10.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb38eec1640 10 monclient(hunting): picked mon.noname-a con 0x7fb3881d74d0 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:10.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb38eec1640 10 monclient(hunting): picked mon.noname-c con 0x7fb388138ca0 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:10.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb38eec1640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3880af860 con 0x7fb3881d74d0 2024-03-29T10:51:10.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb38eec1640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3880b1590 con 0x7fb38809ac40 2024-03-29T10:51:10.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb38cc5d640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fb38809ac40 0x7fb3880b0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:44648/0 (socket says 172.21.15.118:44648) 2024-03-29T10:51:10.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb38cc5d640 1 -- 172.21.15.118:0/610468979 learned_addr learned my addr 172.21.15.118:0/610468979 (peer_addr_for_me v1:172.21.15.118:0/0) 2024-03-29T10:51:10.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb38eec1640 1 -- 172.21.15.118:0/610468979 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3881d8070 con 0x7fb388138ca0 2024-03-29T10:51:10.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb38eec1640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:10.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb38eec1640 10 monclient(hunting): authenticate will time out at 811.228816s 2024-03-29T10:51:10.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb3857fa640 10 client.?.objecter ms_handle_connect 0x7fb38809ac40 2024-03-29T10:51:10.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb3857fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:10.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb3857fa640 10 client.?.objecter ms_handle_connect 0x7fb3881d74d0 2024-03-29T10:51:10.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb3857fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:10.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb3857fa640 10 client.?.objecter ms_handle_connect 0x7fb388138ca0 2024-03-29T10:51:10.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb3857fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:10.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104708533 0 0) 0x7fb3880b1590 con 0x7fb38809ac40 2024-03-29T10:51:10.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb3857fa640 10 monclient(hunting): _negotiate 2024-03-29T10:51:10.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb3857fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:10.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb3857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:10.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb3857fa640 10 monclient(hunting): my global_id is 4123 2024-03-29T10:51:10.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb36c004eb0 con 0x7fb38809ac40 2024-03-29T10:51:10.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3947554110 0 0) 0x7fb3880af860 con 0x7fb3881d74d0 2024-03-29T10:51:10.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb3857fa640 10 monclient(hunting): _negotiate 2024-03-29T10:51:10.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb3857fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:10.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb3857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:10.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb3857fa640 10 monclient(hunting): my global_id is 4116 2024-03-29T10:51:10.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3880b1590 con 0x7fb3881d74d0 2024-03-29T10:51:10.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803017153 0 0) 0x7fb3881d8070 con 0x7fb388138ca0 2024-03-29T10:51:10.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb3857fa640 10 monclient(hunting): _negotiate 2024-03-29T10:51:10.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb3857fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:10.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb3857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:10.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb3857fa640 10 monclient(hunting): my global_id is 4118 2024-03-29T10:51:10.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.910+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3880af860 con 0x7fb388138ca0 2024-03-29T10:51:10.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3669332916 0 0) 0x7fb36c004eb0 con 0x7fb38809ac40 2024-03-29T10:51:10.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3881d8070 con 0x7fb38809ac40 2024-03-29T10:51:10.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb37c001980 con 0x7fb38809ac40 2024-03-29T10:51:10.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:10.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:10.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 10 monclient(hunting): dump: 2024-03-29T10:51:10.909 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:10.909 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:10.909 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.910 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.910 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:10.910 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:10.910 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:10.910 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:10.910 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:10.910 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:10.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:10.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 119297976 0 0) 0x7fb3880b1590 con 0x7fb3881d74d0 2024-03-29T10:51:10.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb36c004eb0 con 0x7fb3881d74d0 2024-03-29T10:51:10.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 360245093 0 0) 0x7fb3880af860 con 0x7fb388138ca0 2024-03-29T10:51:10.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3880b1590 con 0x7fb388138ca0 2024-03-29T10:51:10.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb374001b80 con 0x7fb3881d74d0 2024-03-29T10:51:10.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:10.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:10.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 10 monclient(hunting): dump: 2024-03-29T10:51:10.911 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:10.911 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:10.911 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.911 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.911 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:10.911 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:10.911 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:10.911 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:10.911 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:10.911 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:10.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb380001b40 con 0x7fb388138ca0 2024-03-29T10:51:10.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:10.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:10.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 10 monclient(hunting): dump: 2024-03-29T10:51:10.912 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:10.912 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:10.912 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.912 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.912 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:10.912 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:10.912 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:10.912 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:10.912 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:10.912 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:10.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3308442497 0 0) 0x7fb3881d8070 con 0x7fb38809ac40 2024-03-29T10:51:10.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 >> v1:172.21.15.112:6790/0 conn(0x7fb388138ca0 legacy=0x7fb388139110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:10.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 >> v1:172.21.15.112:6789/0 conn(0x7fb3881d74d0 legacy=0x7fb3881d7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:10.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 10 monclient: _finish_hunting 0 2024-03-29T10:51:10.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 1 monclient: found mon.b 2024-03-29T10:51:10.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:10.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:10.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3880b1800 con 0x7fb38809ac40 2024-03-29T10:51:10.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 10 monclient: _finish_auth 0 2024-03-29T10:51:10.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 10 monclient: _check_auth_tickets 2024-03-29T10:51:10.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:10.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb38eec1640 5 monclient: authenticate success, global_id 4123 2024-03-29T10:51:10.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb38eec1640 10 monclient: _renew_subs 2024-03-29T10:51:10.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb38eec1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:10.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb38eec1640 1 -- 172.21.15.118:0/610468979 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3880af480 con 0x7fb38809ac40 2024-03-29T10:51:10.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb38eec1640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:10.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb38eec1640 10 monclient: _renew_subs 2024-03-29T10:51:10.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb38eec1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:10.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb38eec1640 1 -- 172.21.15.118:0/610468979 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3880b0af0 con 0x7fb38809ac40 2024-03-29T10:51:10.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb38eec1640 1 librados: init done 2024-03-29T10:51:10.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb37c003610 con 0x7fb38809ac40 2024-03-29T10:51:10.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:10.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb37c003960 con 0x7fb38809ac40 2024-03-29T10:51:10.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:10.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb386ffd640 4 set_mon_vals no callback set 2024-03-29T10:51:10.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:10.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 10 monclient: dump: 2024-03-29T10:51:10.914 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:10.914 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:10.914 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.915 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:10.915 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:10.915 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:10.915 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:10.915 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:10.915 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:10.915 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:10.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb37c003e50 con 0x7fb38809ac40 2024-03-29T10:51:10.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 1585165818 0 0) 0x7fb37c006600 con 0x7fb38809ac40 2024-03-29T10:51:10.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 10 client.4123.objecter ms_dispatch 0x7fb388000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T10:51:10.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 3 client.4123.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T10:51:10.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 3 client.4123.objecter handle_osd_map decoding full epoch 3 2024-03-29T10:51:10.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.911+0000 7fb3857fa640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-29T10:51:10.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.912+0000 7fb38eec1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:10.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.912+0000 7fb38eec1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:10.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.912+0000 7fb38eec1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:10.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.912+0000 7fb38eec1640 1 -- 172.21.15.118:0/610468979 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb368005080 con 0x7fb38809ac40 2024-03-29T10:51:10.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.914+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb37c004440 con 0x7fb38809ac40 2024-03-29T10:51:10.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.914+0000 7fb3857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:10.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.914+0000 7fb3857fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:10.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.983+0000 7fb38eec1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "600a2c50-5341-4503-847b-fd2a2922a007", "id": 1}] 2024-03-29T10:51:10.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.983+0000 7fb38eec1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "600a2c50-5341-4503-847b-fd2a2922a007", "id": 1}] 2024-03-29T10:51:10.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.983+0000 7fb38eec1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:10.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.983+0000 7fb38eec1640 1 -- 172.21.15.118:0/610468979 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd new", "uuid": "600a2c50-5341-4503-847b-fd2a2922a007", "id": 1} v 0) v1 -- 0x7fb3680058b0 con 0x7fb38809ac40 2024-03-29T10:51:10.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.986+0000 7fb3857fa640 1 -- 172.21.15.118:0/610468979 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "600a2c50-5341-4503-847b-fd2a2922a007", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2759452080 0 940483276) 0x7fb37c005890 con 0x7fb38809ac40 2024-03-29T10:51:10.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.986+0000 7fb3857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "600a2c50-5341-4503-847b-fd2a2922a007", "id": 1}] 2024-03-29T10:51:10.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.986+0000 7fb3857fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:10.968 INFO:teuthology.orchestra.run.smithi118.stdout:1 2024-03-29T10:51:10.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.987+0000 7fb38eec1640 10 librados: watch_flush enter 2024-03-29T10:51:10.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.987+0000 7fb38eec1640 10 librados: watch_flush exit 2024-03-29T10:51:10.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.987+0000 7fb38eec1640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-29T10:51:10.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.987+0000 7fb38eec1640 10 client.4123.objecter successfully canceled tick 2024-03-29T10:51:10.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.987+0000 7fb38eec1640 10 monclient: shutdown 2024-03-29T10:51:10.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.987+0000 7fb38eec1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:10.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.987+0000 7fb38eec1640 1 -- 172.21.15.118:0/610468979 >> v1:172.21.15.118:6789/0 conn(0x7fb38809ac40 legacy=0x7fb3880b0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:10.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.987+0000 7fb38eec1640 1 -- 172.21.15.118:0/610468979 shutdown_connections 2024-03-29T10:51:10.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.987+0000 7fb38eec1640 1 -- 172.21.15.118:0/610468979 >> 172.21.15.118:0/610468979 conn(0x7fb388066b00 msgr2=0x7fb38813e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:10.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.987+0000 7fb38eec1640 1 -- 172.21.15.118:0/610468979 shutdown_connections 2024-03-29T10:51:10.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.987+0000 7fb38eec1640 1 -- 172.21.15.118:0/610468979 wait complete. 2024-03-29T10:51:10.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:10.987+0000 7fb38eec1640 1 librados: shutdown 2024-03-29T10:51:10.978 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new 3ec4da85-e6cc-4d66-b76e-986e94597743 2 2024-03-29T10:51:11.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.096+0000 7f79c7582640 10 monclient: get_monmap_and_config 2024-03-29T10:51:11.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.096+0000 7f79c7582640 10 monclient: build_initial_monmap 2024-03-29T10:51:11.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.096+0000 7f79c7582640 10 monclient: monmap: 2024-03-29T10:51:11.077 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2024-03-29T10:51:11.077 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.078 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:51:11.096644+0000 2024-03-29T10:51:11.078 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:51:11.096644+0000 2024-03-29T10:51:11.078 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:11.078 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.078 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:11.078 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:51:11.078 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:51:11.078 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.096+0000 7f79c7582640 1 Processor -- start 2024-03-29T10:51:11.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.097+0000 7f79c7582640 1 -- start start 2024-03-29T10:51:11.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.097+0000 7f79c7582640 10 monclient: init 2024-03-29T10:51:11.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.097+0000 7f79c7582640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:11.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.097+0000 7f79c7582640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:51:11.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.097+0000 7f79c7582640 10 monclient(hunting): picked mon.noname-b con 0x7f79c009ac40 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:11.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.097+0000 7f79c7582640 10 monclient(hunting): picked mon.noname-c con 0x7f79c009c870 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:11.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.097+0000 7f79c7582640 10 monclient(hunting): picked mon.noname-a con 0x7f79c00b4360 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:11.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.097+0000 7f79c7582640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79c009dec0 con 0x7f79c00b4360 2024-03-29T10:51:11.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.097+0000 7f79c7582640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79c009da30 con 0x7f79c009c870 2024-03-29T10:51:11.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.097+0000 7f79c7582640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79c00b39c0 con 0x7f79c009ac40 2024-03-29T10:51:11.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.097+0000 7f79c7582640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:11.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.097+0000 7f79c7582640 10 monclient(hunting): authenticate will time out at 811.415838s 2024-03-29T10:51:11.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.097+0000 7f79c4b1d640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f79c009c870 0x7f79c013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:44662/0 (socket says 172.21.15.118:44662) 2024-03-29T10:51:11.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.097+0000 7f79c4b1d640 1 -- 172.21.15.118:0/738615656 learned_addr learned my addr 172.21.15.118:0/738615656 (peer_addr_for_me v1:172.21.15.118:0/0) 2024-03-29T10:51:11.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 1 -- 172.21.15.118:0/738615656 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3282249596 0 0) 0x7f79c009da30 con 0x7f79c009c870 2024-03-29T10:51:11.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 10 monclient(hunting): my global_id is 4126 2024-03-29T10:51:11.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 1 -- 172.21.15.118:0/738615656 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79ac003520 con 0x7f79c009c870 2024-03-29T10:51:11.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 1 -- 172.21.15.118:0/738615656 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459921561 0 0) 0x7f79b8001490 con 0x7f79c00b4360 2024-03-29T10:51:11.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 10 monclient(hunting): my global_id is 4119 2024-03-29T10:51:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 1 -- 172.21.15.118:0/738615656 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79c009da30 con 0x7f79c00b4360 2024-03-29T10:51:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 1 -- 172.21.15.118:0/738615656 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359730921 0 0) 0x7f79c00b39c0 con 0x7f79c009ac40 2024-03-29T10:51:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 10 monclient(hunting): my global_id is 4121 2024-03-29T10:51:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 1 -- 172.21.15.118:0/738615656 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79b8001490 con 0x7f79c009ac40 2024-03-29T10:51:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 1 -- 172.21.15.118:0/738615656 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1179111804 0 0) 0x7f79ac003520 con 0x7f79c009c870 2024-03-29T10:51:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 1 -- 172.21.15.118:0/738615656 >> v1:172.21.15.112:6790/0 conn(0x7f79c009ac40 legacy=0x7f79c00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 1 -- 172.21.15.118:0/738615656 >> v1:172.21.15.112:6789/0 conn(0x7f79c00b4360 legacy=0x7f79c014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:51:11.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 1 monclient: found mon.noname-c 2024-03-29T10:51:11.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:11.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T10:51:11.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 1 -- 172.21.15.118:0/738615656 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c00b5160 con 0x7f79c009c870 2024-03-29T10:51:11.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 10 monclient: _finish_auth 0 2024-03-29T10:51:11.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:51:11.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:11.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79c7582640 5 monclient: authenticate success, global_id 4126 2024-03-29T10:51:11.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 1 -- 172.21.15.118:0/738615656 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f79b0001690 con 0x7f79c009c870 2024-03-29T10:51:11.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:11.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 10 monclient: dump: 2024-03-29T10:51:11.090 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.090 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.091 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.091 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.091 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.091 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.091 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.091 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.091 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.091 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.091 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79c7582640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:11.091 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 1 -- 172.21.15.118:0/738615656 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79b0001870 con 0x7f79c009c870 2024-03-29T10:51:11.091 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:11.091 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79c7582640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:11.091 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 1 -- 172.21.15.118:0/738615656 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f79b0001c80 con 0x7f79c009c870 2024-03-29T10:51:11.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79c7582640 4 set_mon_vals no callback set 2024-03-29T10:51:11.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:11.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79b7fff640 10 monclient: dump: 2024-03-29T10:51:11.092 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.092 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.092 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.092 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.092 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.092 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.092 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.092 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.092 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.092 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79c7582640 10 monclient: shutdown 2024-03-29T10:51:11.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79c7582640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:11.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79c7582640 1 -- 172.21.15.118:0/738615656 >> v1:172.21.15.118:6789/0 conn(0x7f79c009c870 legacy=0x7f79c013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79c7582640 1 -- 172.21.15.118:0/738615656 shutdown_connections 2024-03-29T10:51:11.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.098+0000 7f79c7582640 1 -- 172.21.15.118:0/738615656 >> 172.21.15.118:0/738615656 conn(0x7f79c0066b00 msgr2=0x7f79c0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:11.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.099+0000 7f79c7582640 1 -- 172.21.15.118:0/738615656 shutdown_connections 2024-03-29T10:51:11.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.099+0000 7f79c7582640 1 -- 172.21.15.118:0/738615656 wait complete. 2024-03-29T10:51:11.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.099+0000 7f79c7582640 10 monclient: build_initial_monmap 2024-03-29T10:51:11.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.099+0000 7f79c7582640 10 monclient: monmap: 2024-03-29T10:51:11.093 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2024-03-29T10:51:11.093 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:11.093 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 0.000000 2024-03-29T10:51:11.093 INFO:teuthology.orchestra.run.smithi118.stderr:created 0.000000 2024-03-29T10:51:11.093 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:11.094 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.094 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:11.094 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:51:11.094 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:51:11.094 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.099+0000 7f79c7582640 1 librados: starting msgr at 2024-03-29T10:51:11.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.099+0000 7f79c7582640 1 librados: starting objecter 2024-03-29T10:51:11.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.099+0000 7f79c7582640 1 Processor -- start 2024-03-29T10:51:11.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.099+0000 7f79c7582640 1 -- start start 2024-03-29T10:51:11.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.099+0000 7f79c7582640 1 librados: setting wanted keys 2024-03-29T10:51:11.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.099+0000 7f79c7582640 1 librados: calling monclient init 2024-03-29T10:51:11.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.099+0000 7f79c7582640 10 monclient: init 2024-03-29T10:51:11.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79c7582640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:11.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79c7582640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:51:11.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79c7582640 10 monclient(hunting): picked mon.noname-b con 0x7f79c009ac40 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:11.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79c7582640 10 monclient(hunting): picked mon.noname-c con 0x7f79c0140d40 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:11.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79c7582640 10 monclient(hunting): picked mon.noname-a con 0x7f79c01402a0 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:11.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79c7582640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79c013a7d0 con 0x7f79c01402a0 2024-03-29T10:51:11.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79c7582640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79c00b7780 con 0x7f79c009ac40 2024-03-29T10:51:11.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79c7582640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79c013ffa0 con 0x7f79c0140d40 2024-03-29T10:51:11.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79c7582640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:11.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79c7582640 10 monclient(hunting): authenticate will time out at 811.418643s 2024-03-29T10:51:11.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79c531e640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f79c009ac40 0x7f79c0142420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:44670/0 (socket says 172.21.15.118:44670) 2024-03-29T10:51:11.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79c531e640 1 -- 172.21.15.118:0/2638881031 learned_addr learned my addr 172.21.15.118:0/2638881031 (peer_addr_for_me v1:172.21.15.118:0/0) 2024-03-29T10:51:11.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 10 client.?.objecter ms_handle_connect 0x7f79c009ac40 2024-03-29T10:51:11.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:11.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 10 client.?.objecter ms_handle_connect 0x7f79c0140d40 2024-03-29T10:51:11.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:11.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053590304 0 0) 0x7f79c00b7780 con 0x7f79c009ac40 2024-03-29T10:51:11.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 10 monclient(hunting): my global_id is 4129 2024-03-29T10:51:11.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79a4004eb0 con 0x7f79c009ac40 2024-03-29T10:51:11.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 10 client.?.objecter ms_handle_connect 0x7f79c01402a0 2024-03-29T10:51:11.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:11.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146796639 0 0) 0x7f79c013ffa0 con 0x7f79c0140d40 2024-03-29T10:51:11.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 10 monclient(hunting): my global_id is 4124 2024-03-29T10:51:11.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79c00b7780 con 0x7f79c0140d40 2024-03-29T10:51:11.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031965171 0 0) 0x7f79c013a7d0 con 0x7f79c01402a0 2024-03-29T10:51:11.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.097 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.097 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.097 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 10 monclient(hunting): my global_id is 4122 2024-03-29T10:51:11.097 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79c013ffa0 con 0x7f79c01402a0 2024-03-29T10:51:11.097 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3717565361 0 0) 0x7f79a4004eb0 con 0x7f79c009ac40 2024-03-29T10:51:11.097 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79c013a7d0 con 0x7f79c009ac40 2024-03-29T10:51:11.097 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.100+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f79bc001930 con 0x7f79c009ac40 2024-03-29T10:51:11.097 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.097 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:11.097 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 10 monclient(hunting): dump: 2024-03-29T10:51:11.097 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.097 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.097 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.097 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.097 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.098 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.098 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.098 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.098 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.098 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.098 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:11.098 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3586845089 0 0) 0x7f79c00b7780 con 0x7f79c0140d40 2024-03-29T10:51:11.098 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79a4004eb0 con 0x7f79c0140d40 2024-03-29T10:51:11.098 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f79b0001b80 con 0x7f79c0140d40 2024-03-29T10:51:11.098 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.098 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:11.098 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 10 monclient(hunting): dump: 2024-03-29T10:51:11.098 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.098 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.099 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.099 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.099 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.099 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.099 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.099 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.099 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.099 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.099 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1552336372 0 0) 0x7f79c013ffa0 con 0x7f79c01402a0 2024-03-29T10:51:11.099 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79c00b7780 con 0x7f79c01402a0 2024-03-29T10:51:11.099 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f79b8001bc0 con 0x7f79c01402a0 2024-03-29T10:51:11.099 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.099 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:11.099 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 10 monclient(hunting): dump: 2024-03-29T10:51:11.100 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.100 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.100 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.100 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.100 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.100 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.100 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.100 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.100 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.100 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2364596602 0 0) 0x7f79c013a7d0 con 0x7f79c009ac40 2024-03-29T10:51:11.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 >> v1:172.21.15.112:6790/0 conn(0x7f79c0140d40 legacy=0x7f79c01399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 >> v1:172.21.15.112:6789/0 conn(0x7f79c01402a0 legacy=0x7f79c013a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:51:11.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 1 monclient: found mon.b 2024-03-29T10:51:11.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:11.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:11.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c01c9c80 con 0x7f79c009ac40 2024-03-29T10:51:11.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:51:11.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:51:11.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:11.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79c7582640 5 monclient: authenticate success, global_id 4129 2024-03-29T10:51:11.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79c7582640 10 monclient: _renew_subs 2024-03-29T10:51:11.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79c7582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:11.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79c7582640 1 -- 172.21.15.118:0/2638881031 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79c01c9ed0 con 0x7f79c009ac40 2024-03-29T10:51:11.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79bc003610 con 0x7f79c009ac40 2024-03-29T10:51:11.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:11.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f79bc003960 con 0x7f79c009ac40 2024-03-29T10:51:11.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:11.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 10 monclient: dump: 2024-03-29T10:51:11.102 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.102 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.102 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.102 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.102 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.102 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.102 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.102 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.102 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.102 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79c7582640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:11.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79c7582640 10 monclient: _renew_subs 2024-03-29T10:51:11.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b77fe640 4 set_mon_vals no callback set 2024-03-29T10:51:11.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79c7582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:11.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79c7582640 1 -- 172.21.15.118:0/2638881031 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79c0142d10 con 0x7f79c009ac40 2024-03-29T10:51:11.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79c7582640 1 librados: init done 2024-03-29T10:51:11.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f79bc003ed0 con 0x7f79c009ac40 2024-03-29T10:51:11.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2976495529 0 0) 0x7f79bc0057c0 con 0x7f79c009ac40 2024-03-29T10:51:11.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 10 client.4129.objecter ms_dispatch 0x7f79c0000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T10:51:11.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 3 client.4129.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T10:51:11.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 3 client.4129.objecter handle_osd_map decoding full epoch 4 2024-03-29T10:51:11.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.101+0000 7f79b5ffb640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-29T10:51:11.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.102+0000 7f79c7582640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:11.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.102+0000 7f79c7582640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:11.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.102+0000 7f79c7582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:11.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.102+0000 7f79c7582640 1 -- 172.21.15.118:0/2638881031 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79c009b020 con 0x7f79c009ac40 2024-03-29T10:51:11.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.104+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f79bc005980 con 0x7f79c009ac40 2024-03-29T10:51:11.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.104+0000 7f79b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:11.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.104+0000 7f79b5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:11.112 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:11.131+0000 7f0c627b8040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:51:11.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.174+0000 7f79c7582640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3ec4da85-e6cc-4d66-b76e-986e94597743", "id": 2}] 2024-03-29T10:51:11.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.174+0000 7f79c7582640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3ec4da85-e6cc-4d66-b76e-986e94597743", "id": 2}] 2024-03-29T10:51:11.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.174+0000 7f79c7582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:11.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.174+0000 7f79c7582640 1 -- 172.21.15.118:0/2638881031 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3ec4da85-e6cc-4d66-b76e-986e94597743", "id": 2} v 0) v1 -- 0x7f79c004f730 con 0x7f79c009ac40 2024-03-29T10:51:11.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.177+0000 7f79b5ffb640 1 -- 172.21.15.118:0/2638881031 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3ec4da85-e6cc-4d66-b76e-986e94597743", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 312393534 0 216599125) 0x7f79bc005b60 con 0x7f79c009ac40 2024-03-29T10:51:11.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.177+0000 7f79b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3ec4da85-e6cc-4d66-b76e-986e94597743", "id": 2}] 2024-03-29T10:51:11.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.177+0000 7f79b5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:11.158 INFO:teuthology.orchestra.run.smithi118.stdout:2 2024-03-29T10:51:11.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.178+0000 7f79c7582640 10 librados: watch_flush enter 2024-03-29T10:51:11.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.178+0000 7f79c7582640 10 librados: watch_flush exit 2024-03-29T10:51:11.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.178+0000 7f79c7582640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-29T10:51:11.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.178+0000 7f79c7582640 10 client.4129.objecter successfully canceled tick 2024-03-29T10:51:11.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.178+0000 7f79c7582640 10 monclient: shutdown 2024-03-29T10:51:11.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.178+0000 7f79c7582640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:11.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.178+0000 7f79c7582640 1 -- 172.21.15.118:0/2638881031 >> v1:172.21.15.118:6789/0 conn(0x7f79c009ac40 legacy=0x7f79c0142420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.178+0000 7f79c7582640 1 -- 172.21.15.118:0/2638881031 shutdown_connections 2024-03-29T10:51:11.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.178+0000 7f79c7582640 1 -- 172.21.15.118:0/2638881031 >> 172.21.15.118:0/2638881031 conn(0x7f79c0066b00 msgr2=0x7f79c013e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:11.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.178+0000 7f79c7582640 1 -- 172.21.15.118:0/2638881031 shutdown_connections 2024-03-29T10:51:11.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.178+0000 7f79c7582640 1 -- 172.21.15.118:0/2638881031 wait complete. 2024-03-29T10:51:11.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.178+0000 7f79c7582640 1 librados: shutdown 2024-03-29T10:51:11.166 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:11.185+0000 7f0c627b8040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:51:11.170 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new bcf148cb-34a5-47b1-9c89-9956c132153b 3 2024-03-29T10:51:11.211 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:11.229+0000 7f0c627b8040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:51:11.235 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:11.253+0000 7f03b67b6040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:51:11.263 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:11.282+0000 7f0c627b8040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:51:11.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.286+0000 7f39e039d640 10 monclient: get_monmap_and_config 2024-03-29T10:51:11.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.286+0000 7f39e039d640 10 monclient: build_initial_monmap 2024-03-29T10:51:11.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.286+0000 7f39e039d640 10 monclient: monmap: 2024-03-29T10:51:11.268 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2024-03-29T10:51:11.269 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.269 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:51:11.288085+0000 2024-03-29T10:51:11.269 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:51:11.288085+0000 2024-03-29T10:51:11.269 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:11.269 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.269 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:11.269 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:51:11.269 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:51:11.269 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.287+0000 7f39e039d640 1 Processor -- start 2024-03-29T10:51:11.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.287+0000 7f39e039d640 1 -- start start 2024-03-29T10:51:11.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.287+0000 7f39e039d640 10 monclient: init 2024-03-29T10:51:11.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39e039d640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:11.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39e039d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:51:11.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39e039d640 10 monclient(hunting): picked mon.noname-a con 0x7f39d809ac40 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:11.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39e039d640 10 monclient(hunting): picked mon.noname-b con 0x7f39d809c870 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:11.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39e039d640 10 monclient(hunting): picked mon.noname-c con 0x7f39d80b4360 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:11.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39e039d640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39d809dec0 con 0x7f39d809ac40 2024-03-29T10:51:11.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39e039d640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39d809da30 con 0x7f39d80b4360 2024-03-29T10:51:11.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39e039d640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39d80b39c0 con 0x7f39d809c870 2024-03-29T10:51:11.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39e039d640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:11.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39e039d640 10 monclient(hunting): authenticate will time out at 811.607298s 2024-03-29T10:51:11.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39de93a640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f39d80b4360 0x7f39d814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:44680/0 (socket says 172.21.15.118:44680) 2024-03-29T10:51:11.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39de93a640 1 -- 172.21.15.118:0/1004656822 learned_addr learned my addr 172.21.15.118:0/1004656822 (peer_addr_for_me v1:172.21.15.118:0/0) 2024-03-29T10:51:11.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 1 -- 172.21.15.118:0/1004656822 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290006080 0 0) 0x7f39d809da30 con 0x7f39d80b4360 2024-03-29T10:51:11.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 10 monclient(hunting): my global_id is 4132 2024-03-29T10:51:11.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 1 -- 172.21.15.118:0/1004656822 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39c4003520 con 0x7f39d80b4360 2024-03-29T10:51:11.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 1 -- 172.21.15.118:0/1004656822 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30438394 0 0) 0x7f39c8001110 con 0x7f39d809ac40 2024-03-29T10:51:11.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 10 monclient(hunting): my global_id is 4125 2024-03-29T10:51:11.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 1 -- 172.21.15.118:0/1004656822 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39d809da30 con 0x7f39d809ac40 2024-03-29T10:51:11.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 1 -- 172.21.15.118:0/1004656822 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3140876272 0 0) 0x7f39d80b39c0 con 0x7f39d809c870 2024-03-29T10:51:11.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 10 monclient(hunting): my global_id is 4127 2024-03-29T10:51:11.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 1 -- 172.21.15.118:0/1004656822 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39c8001110 con 0x7f39d809c870 2024-03-29T10:51:11.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 1 -- 172.21.15.118:0/1004656822 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 337554778 0 0) 0x7f39c4003520 con 0x7f39d80b4360 2024-03-29T10:51:11.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 1 -- 172.21.15.118:0/1004656822 >> v1:172.21.15.112:6790/0 conn(0x7f39d809c870 legacy=0x7f39d813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 1 -- 172.21.15.118:0/1004656822 >> v1:172.21.15.112:6789/0 conn(0x7f39d809ac40 legacy=0x7f39d80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 10 monclient: _finish_hunting 0 2024-03-29T10:51:11.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 1 monclient: found mon.noname-c 2024-03-29T10:51:11.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:11.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T10:51:11.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 1 -- 172.21.15.118:0/1004656822 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d80b5160 con 0x7f39d80b4360 2024-03-29T10:51:11.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 10 monclient: _finish_auth 0 2024-03-29T10:51:11.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 10 monclient: _check_auth_tickets 2024-03-29T10:51:11.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:11.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 1 -- 172.21.15.118:0/1004656822 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f39d0001a10 con 0x7f39d80b4360 2024-03-29T10:51:11.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39e039d640 5 monclient: authenticate success, global_id 4132 2024-03-29T10:51:11.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39e039d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:11.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:11.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 10 monclient: dump: 2024-03-29T10:51:11.281 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.282 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.282 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.282 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.282 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.282 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.282 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.282 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.282 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.282 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39e039d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:11.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 1 -- 172.21.15.118:0/1004656822 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39d0001bf0 con 0x7f39d80b4360 2024-03-29T10:51:11.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:11.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 1 -- 172.21.15.118:0/1004656822 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f39d0002000 con 0x7f39d80b4360 2024-03-29T10:51:11.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39e039d640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:11.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39e039d640 4 set_mon_vals no callback set 2024-03-29T10:51:11.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:11.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.288+0000 7f39dd137640 10 monclient: dump: 2024-03-29T10:51:11.283 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.283 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.283 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.283 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.283 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.283 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.283 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.283 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.283 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.283 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.289+0000 7f39e039d640 10 monclient: shutdown 2024-03-29T10:51:11.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.289+0000 7f39e039d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:11.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.289+0000 7f39e039d640 1 -- 172.21.15.118:0/1004656822 >> v1:172.21.15.118:6789/0 conn(0x7f39d80b4360 legacy=0x7f39d814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.289+0000 7f39e039d640 1 -- 172.21.15.118:0/1004656822 shutdown_connections 2024-03-29T10:51:11.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.289+0000 7f39e039d640 1 -- 172.21.15.118:0/1004656822 >> 172.21.15.118:0/1004656822 conn(0x7f39d8066b00 msgr2=0x7f39d8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:11.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.289+0000 7f39e039d640 1 -- 172.21.15.118:0/1004656822 shutdown_connections 2024-03-29T10:51:11.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.289+0000 7f39e039d640 1 -- 172.21.15.118:0/1004656822 wait complete. 2024-03-29T10:51:11.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.290+0000 7f39e039d640 10 monclient: build_initial_monmap 2024-03-29T10:51:11.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.290+0000 7f39e039d640 10 monclient: monmap: 2024-03-29T10:51:11.284 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2024-03-29T10:51:11.284 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:11.284 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 0.000000 2024-03-29T10:51:11.284 INFO:teuthology.orchestra.run.smithi118.stderr:created 0.000000 2024-03-29T10:51:11.284 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:11.284 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.284 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:51:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:51:11.285 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.290+0000 7f39e039d640 1 librados: starting msgr at 2024-03-29T10:51:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.290+0000 7f39e039d640 1 librados: starting objecter 2024-03-29T10:51:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.290+0000 7f39e039d640 1 Processor -- start 2024-03-29T10:51:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.290+0000 7f39e039d640 1 -- start start 2024-03-29T10:51:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.290+0000 7f39e039d640 1 librados: setting wanted keys 2024-03-29T10:51:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.290+0000 7f39e039d640 1 librados: calling monclient init 2024-03-29T10:51:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.290+0000 7f39e039d640 10 monclient: init 2024-03-29T10:51:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39e039d640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39e039d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:51:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39e039d640 10 monclient(hunting): picked mon.noname-c con 0x7f39d809ac40 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39e039d640 10 monclient(hunting): picked mon.noname-b con 0x7f39d80b63e0 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39e039d640 10 monclient(hunting): picked mon.noname-a con 0x7f39d81461c0 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39e039d640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39d80b7520 con 0x7f39d81461c0 2024-03-29T10:51:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39e039d640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39d80b8960 con 0x7f39d80b63e0 2024-03-29T10:51:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39e039d640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39d80b7d30 con 0x7f39d809ac40 2024-03-29T10:51:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39e039d640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39e039d640 10 monclient(hunting): authenticate will time out at 811.610318s 2024-03-29T10:51:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39dd938640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f39d80b63e0 0x7f39d81d4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:44690/0 (socket says 172.21.15.118:44690) 2024-03-29T10:51:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39dd938640 1 -- 172.21.15.118:0/4042209013 learned_addr learned my addr 172.21.15.118:0/4042209013 (peer_addr_for_me v1:172.21.15.118:0/0) 2024-03-29T10:51:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 client.?.objecter ms_handle_connect 0x7f39d80b63e0 2024-03-29T10:51:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 client.?.objecter ms_handle_connect 0x7f39d809ac40 2024-03-29T10:51:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 client.?.objecter ms_handle_connect 0x7f39d81461c0 2024-03-29T10:51:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115512152 0 0) 0x7f39d80b7520 con 0x7f39d81461c0 2024-03-29T10:51:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient(hunting): my global_id is 4128 2024-03-29T10:51:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39bc004eb0 con 0x7f39d81461c0 2024-03-29T10:51:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851733609 0 0) 0x7f39d80b8960 con 0x7f39d80b63e0 2024-03-29T10:51:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient(hunting): my global_id is 4135 2024-03-29T10:51:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39d80b7520 con 0x7f39d80b63e0 2024-03-29T10:51:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 210307903 0 0) 0x7f39d80b7d30 con 0x7f39d809ac40 2024-03-29T10:51:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient(hunting): my global_id is 4130 2024-03-29T10:51:11.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39d80b8960 con 0x7f39d809ac40 2024-03-29T10:51:11.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3695795566 0 0) 0x7f39bc004eb0 con 0x7f39d81461c0 2024-03-29T10:51:11.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39d80b7d30 con 0x7f39d81461c0 2024-03-29T10:51:11.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f39d0001f00 con 0x7f39d81461c0 2024-03-29T10:51:11.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:11.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient(hunting): dump: 2024-03-29T10:51:11.288 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.288 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.288 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.288 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.288 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.289 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.289 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.289 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.289 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.289 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:11.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4132898038 0 0) 0x7f39d80b7520 con 0x7f39d80b63e0 2024-03-29T10:51:11.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39bc004eb0 con 0x7f39d80b63e0 2024-03-29T10:51:11.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f39d4001880 con 0x7f39d80b63e0 2024-03-29T10:51:11.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:11.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient(hunting): dump: 2024-03-29T10:51:11.289 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.289 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.289 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.289 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.290 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.290 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.290 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.290 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.290 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.290 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 586893032 0 0) 0x7f39d80b8960 con 0x7f39d809ac40 2024-03-29T10:51:11.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39d80b7520 con 0x7f39d809ac40 2024-03-29T10:51:11.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f39c80018a0 con 0x7f39d809ac40 2024-03-29T10:51:11.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:11.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient(hunting): dump: 2024-03-29T10:51:11.290 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.291 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.291 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.291 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.291 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.291 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.291 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.291 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.291 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.291 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 129941006 0 0) 0x7f39d80b7d30 con 0x7f39d81461c0 2024-03-29T10:51:11.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 >> v1:172.21.15.112:6790/0 conn(0x7f39d809ac40 legacy=0x7f39d80b8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 >> v1:172.21.15.118:6789/0 conn(0x7f39d80b63e0 legacy=0x7f39d81d4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:11.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 1 monclient: found mon.a 2024-03-29T10:51:11.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:11.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:51:11.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d81d7470 con 0x7f39d81461c0 2024-03-29T10:51:11.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient: _finish_auth 0 2024-03-29T10:51:11.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:11.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:11.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39e039d640 5 monclient: authenticate success, global_id 4128 2024-03-29T10:51:11.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39e039d640 10 monclient: _renew_subs 2024-03-29T10:51:11.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39e039d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:51:11.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.291+0000 7f39e039d640 1 -- 172.21.15.118:0/4042209013 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39d81d76e0 con 0x7f39d81461c0 2024-03-29T10:51:11.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.292+0000 7f39e039d640 10 client.4128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:11.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.292+0000 7f39e039d640 10 monclient: _renew_subs 2024-03-29T10:51:11.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.292+0000 7f39e039d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:51:11.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.292+0000 7f39e039d640 1 -- 172.21.15.118:0/4042209013 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39d81d4e20 con 0x7f39d81461c0 2024-03-29T10:51:11.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.292+0000 7f39e039d640 1 librados: init done 2024-03-29T10:51:11.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.292+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39d0001420 con 0x7f39d81461c0 2024-03-29T10:51:11.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.292+0000 7f39ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:11.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.292+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f39d0004fb0 con 0x7f39d81461c0 2024-03-29T10:51:11.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.292+0000 7f39ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.292+0000 7f39ceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:11.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.292+0000 7f39dc936640 4 set_mon_vals no callback set 2024-03-29T10:51:11.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.292+0000 7f39ceffd640 10 monclient: dump: 2024-03-29T10:51:11.293 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.293 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.293 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.293 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.293 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.293 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.293 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.294 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.294 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.294 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.292+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f39d00054a0 con 0x7f39d81461c0 2024-03-29T10:51:11.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.292+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 1169006798 0 0) 0x7f39d0006ac0 con 0x7f39d81461c0 2024-03-29T10:51:11.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.292+0000 7f39ceffd640 10 client.4128.objecter ms_dispatch 0x7f39d8000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T10:51:11.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.292+0000 7f39ceffd640 3 client.4128.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T10:51:11.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.292+0000 7f39ceffd640 3 client.4128.objecter handle_osd_map decoding full epoch 5 2024-03-29T10:51:11.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.292+0000 7f39ceffd640 20 client.4128.objecter dump_active .. 0 homeless 2024-03-29T10:51:11.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.292+0000 7f39e039d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:11.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.292+0000 7f39e039d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:11.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.292+0000 7f39e039d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:51:11.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.292+0000 7f39e039d640 1 -- 172.21.15.118:0/4042209013 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39c0005080 con 0x7f39d81461c0 2024-03-29T10:51:11.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.295+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f39d00066e0 con 0x7f39d81461c0 2024-03-29T10:51:11.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.295+0000 7f39ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:11.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.295+0000 7f39ceffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:11.295 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:11.310+0000 7f03b67b6040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:51:11.307 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:11.325+0000 7f0c627b8040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:51:11.336 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:11.354+0000 7f03b67b6040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:51:11.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.364+0000 7f39e039d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "bcf148cb-34a5-47b1-9c89-9956c132153b", "id": 3}] 2024-03-29T10:51:11.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.364+0000 7f39e039d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "bcf148cb-34a5-47b1-9c89-9956c132153b", "id": 3}] 2024-03-29T10:51:11.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.364+0000 7f39e039d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:51:11.347 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.364+0000 7f39e039d640 1 -- 172.21.15.118:0/4042209013 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd new", "uuid": "bcf148cb-34a5-47b1-9c89-9956c132153b", "id": 3} v 0) v1 -- 0x7f39c00058b0 con 0x7f39d81461c0 2024-03-29T10:51:11.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.366+0000 7f39ceffd640 1 -- 172.21.15.118:0/4042209013 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bcf148cb-34a5-47b1-9c89-9956c132153b", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2331567038 0 525046306) 0x7f39d0005930 con 0x7f39d81461c0 2024-03-29T10:51:11.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.366+0000 7f39ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "bcf148cb-34a5-47b1-9c89-9956c132153b", "id": 3}] 2024-03-29T10:51:11.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.366+0000 7f39ceffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:11.348 INFO:teuthology.orchestra.run.smithi118.stdout:3 2024-03-29T10:51:11.349 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.367+0000 7f39e039d640 10 librados: watch_flush enter 2024-03-29T10:51:11.349 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.367+0000 7f39e039d640 10 librados: watch_flush exit 2024-03-29T10:51:11.350 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.367+0000 7f39e039d640 20 client.4128.objecter shutdown clearing up homeless session... 2024-03-29T10:51:11.350 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.367+0000 7f39e039d640 10 client.4128.objecter successfully canceled tick 2024-03-29T10:51:11.350 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.367+0000 7f39e039d640 10 monclient: shutdown 2024-03-29T10:51:11.350 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.367+0000 7f39e039d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:11.350 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.367+0000 7f39e039d640 1 -- 172.21.15.118:0/4042209013 >> v1:172.21.15.112:6789/0 conn(0x7f39d81461c0 legacy=0x7f39d81e2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.350 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.367+0000 7f39e039d640 1 -- 172.21.15.118:0/4042209013 shutdown_connections 2024-03-29T10:51:11.350 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.367+0000 7f39e039d640 1 -- 172.21.15.118:0/4042209013 >> 172.21.15.118:0/4042209013 conn(0x7f39d8066b00 msgr2=0x7f39d813d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:11.350 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.367+0000 7f39e039d640 1 -- 172.21.15.118:0/4042209013 shutdown_connections 2024-03-29T10:51:11.350 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.367+0000 7f39e039d640 1 -- 172.21.15.118:0/4042209013 wait complete. 2024-03-29T10:51:11.350 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.367+0000 7f39e039d640 1 librados: shutdown 2024-03-29T10:51:11.359 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new 60b98350-a43a-40a3-8d6a-fa227192c87a 4 2024-03-29T10:51:11.394 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:11.412+0000 7f03b67b6040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:51:11.422 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:11.440+0000 7f0c627b8040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:51:11.438 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:11.456+0000 7f03b67b6040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:51:11.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.475+0000 7ff9abd96640 10 monclient: get_monmap_and_config 2024-03-29T10:51:11.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.475+0000 7ff9abd96640 10 monclient: build_initial_monmap 2024-03-29T10:51:11.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.475+0000 7ff9abd96640 10 monclient: monmap: 2024-03-29T10:51:11.457 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2024-03-29T10:51:11.457 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.457 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:51:11.477104+0000 2024-03-29T10:51:11.457 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:51:11.477104+0000 2024-03-29T10:51:11.457 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:11.458 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.458 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:11.458 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:51:11.458 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:51:11.458 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.476+0000 7ff9abd96640 1 Processor -- start 2024-03-29T10:51:11.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.476+0000 7ff9abd96640 1 -- start start 2024-03-29T10:51:11.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.476+0000 7ff9abd96640 10 monclient: init 2024-03-29T10:51:11.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9abd96640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:11.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9abd96640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:11.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9abd96640 10 monclient(hunting): picked mon.noname-b con 0x7ff9a409ac40 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:11.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9abd96640 10 monclient(hunting): picked mon.noname-a con 0x7ff9a409c870 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:11.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9abd96640 10 monclient(hunting): picked mon.noname-c con 0x7ff9a40b4360 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:11.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9abd96640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a409dec0 con 0x7ff9a409c870 2024-03-29T10:51:11.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9abd96640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a409da30 con 0x7ff9a40b4360 2024-03-29T10:51:11.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9abd96640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a40b39c0 con 0x7ff9a409ac40 2024-03-29T10:51:11.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9abd96640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:11.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9abd96640 10 monclient(hunting): authenticate will time out at 811.796433s 2024-03-29T10:51:11.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9aa333640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7ff9a40b4360 0x7ff9a414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:44692/0 (socket says 172.21.15.118:44692) 2024-03-29T10:51:11.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9aa333640 1 -- 172.21.15.118:0/870667781 learned_addr learned my addr 172.21.15.118:0/870667781 (peer_addr_for_me v1:172.21.15.118:0/0) 2024-03-29T10:51:11.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 1 -- 172.21.15.118:0/870667781 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 361210560 0 0) 0x7ff9a409da30 con 0x7ff9a40b4360 2024-03-29T10:51:11.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 10 monclient(hunting): my global_id is 4138 2024-03-29T10:51:11.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 1 -- 172.21.15.118:0/870667781 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff990003520 con 0x7ff9a40b4360 2024-03-29T10:51:11.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 1 -- 172.21.15.118:0/870667781 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281582317 0 0) 0x7ff9a0001110 con 0x7ff9a409c870 2024-03-29T10:51:11.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 10 monclient(hunting): my global_id is 4131 2024-03-29T10:51:11.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 1 -- 172.21.15.118:0/870667781 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9a409da30 con 0x7ff9a409c870 2024-03-29T10:51:11.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 1 -- 172.21.15.118:0/870667781 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283441033 0 0) 0x7ff9a40b39c0 con 0x7ff9a409ac40 2024-03-29T10:51:11.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 10 monclient(hunting): my global_id is 4133 2024-03-29T10:51:11.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 1 -- 172.21.15.118:0/870667781 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9a0001110 con 0x7ff9a409ac40 2024-03-29T10:51:11.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 1 -- 172.21.15.118:0/870667781 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 603618654 0 0) 0x7ff990003520 con 0x7ff9a40b4360 2024-03-29T10:51:11.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 1 -- 172.21.15.118:0/870667781 >> v1:172.21.15.112:6790/0 conn(0x7ff9a409ac40 legacy=0x7ff9a40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 1 -- 172.21.15.118:0/870667781 >> v1:172.21.15.112:6789/0 conn(0x7ff9a409c870 legacy=0x7ff9a413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 10 monclient: _finish_hunting 0 2024-03-29T10:51:11.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 1 monclient: found mon.noname-c 2024-03-29T10:51:11.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:11.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T10:51:11.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 1 -- 172.21.15.118:0/870667781 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a40b5160 con 0x7ff9a40b4360 2024-03-29T10:51:11.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 10 monclient: _finish_auth 0 2024-03-29T10:51:11.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 10 monclient: _check_auth_tickets 2024-03-29T10:51:11.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:11.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 1 -- 172.21.15.118:0/870667781 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7ff99c001a10 con 0x7ff9a40b4360 2024-03-29T10:51:11.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9abd96640 5 monclient: authenticate success, global_id 4138 2024-03-29T10:51:11.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9abd96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:11.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:11.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 10 monclient: dump: 2024-03-29T10:51:11.470 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.470 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.470 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.470 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.470 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.470 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.470 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.470 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.470 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.471 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9abd96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:11.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 1 -- 172.21.15.118:0/870667781 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff99c001bf0 con 0x7ff9a40b4360 2024-03-29T10:51:11.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:11.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 1 -- 172.21.15.118:0/870667781 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7ff99c002000 con 0x7ff9a40b4360 2024-03-29T10:51:11.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9abd96640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:11.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9abd96640 4 set_mon_vals no callback set 2024-03-29T10:51:11.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:11.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.477+0000 7ff9a8b30640 10 monclient: dump: 2024-03-29T10:51:11.471 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.471 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.471 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.472 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.472 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.472 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.472 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.472 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.472 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.472 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.478+0000 7ff9abd96640 10 monclient: shutdown 2024-03-29T10:51:11.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.478+0000 7ff9abd96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:11.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.478+0000 7ff9abd96640 1 -- 172.21.15.118:0/870667781 >> v1:172.21.15.118:6789/0 conn(0x7ff9a40b4360 legacy=0x7ff9a414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.478+0000 7ff9abd96640 1 -- 172.21.15.118:0/870667781 shutdown_connections 2024-03-29T10:51:11.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.478+0000 7ff9abd96640 1 -- 172.21.15.118:0/870667781 >> 172.21.15.118:0/870667781 conn(0x7ff9a4066b00 msgr2=0x7ff9a4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:11.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.478+0000 7ff9abd96640 1 -- 172.21.15.118:0/870667781 shutdown_connections 2024-03-29T10:51:11.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.478+0000 7ff9abd96640 1 -- 172.21.15.118:0/870667781 wait complete. 2024-03-29T10:51:11.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.479+0000 7ff9abd96640 10 monclient: build_initial_monmap 2024-03-29T10:51:11.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.479+0000 7ff9abd96640 10 monclient: monmap: 2024-03-29T10:51:11.473 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2024-03-29T10:51:11.473 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:11.473 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 0.000000 2024-03-29T10:51:11.473 INFO:teuthology.orchestra.run.smithi118.stderr:created 0.000000 2024-03-29T10:51:11.473 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:11.473 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.473 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:11.473 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:51:11.473 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:51:11.473 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.479+0000 7ff9abd96640 1 librados: starting msgr at 2024-03-29T10:51:11.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.479+0000 7ff9abd96640 1 librados: starting objecter 2024-03-29T10:51:11.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.479+0000 7ff9abd96640 1 Processor -- start 2024-03-29T10:51:11.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.479+0000 7ff9abd96640 1 -- start start 2024-03-29T10:51:11.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.479+0000 7ff9abd96640 1 librados: setting wanted keys 2024-03-29T10:51:11.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.479+0000 7ff9abd96640 1 librados: calling monclient init 2024-03-29T10:51:11.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.479+0000 7ff9abd96640 10 monclient: init 2024-03-29T10:51:11.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff9abd96640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:11.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff9abd96640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:11.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff9abd96640 10 monclient(hunting): picked mon.noname-c con 0x7ff9a409ac40 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:11.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff9abd96640 10 monclient(hunting): picked mon.noname-a con 0x7ff9a4140a10 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:11.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff9abd96640 10 monclient(hunting): picked mon.noname-b con 0x7ff9a41d7e70 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:11.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff9abd96640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a41d7280 con 0x7ff9a4140a10 2024-03-29T10:51:11.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff9abd96640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a4142b20 con 0x7ff9a41d7e70 2024-03-29T10:51:11.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff9abd96640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a4144420 con 0x7ff9a409ac40 2024-03-29T10:51:11.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff9abd96640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:11.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff9abd96640 10 monclient(hunting): authenticate will time out at 811.799398s 2024-03-29T10:51:11.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff9aa333640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7ff9a41d7e70 0x7ff9a40b0f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:44702/0 (socket says 172.21.15.118:44702) 2024-03-29T10:51:11.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff9aa333640 1 -- 172.21.15.118:0/1403826602 learned_addr learned my addr 172.21.15.118:0/1403826602 (peer_addr_for_me v1:172.21.15.118:0/0) 2024-03-29T10:51:11.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 client.?.objecter ms_handle_connect 0x7ff9a41d7e70 2024-03-29T10:51:11.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:11.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 client.?.objecter ms_handle_connect 0x7ff9a4140a10 2024-03-29T10:51:11.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:11.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 client.?.objecter ms_handle_connect 0x7ff9a409ac40 2024-03-29T10:51:11.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:11.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801851043 0 0) 0x7ff9a4142b20 con 0x7ff9a41d7e70 2024-03-29T10:51:11.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 monclient(hunting): my global_id is 4141 2024-03-29T10:51:11.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff988004eb0 con 0x7ff9a41d7e70 2024-03-29T10:51:11.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764110589 0 0) 0x7ff9a41d7280 con 0x7ff9a4140a10 2024-03-29T10:51:11.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 monclient(hunting): my global_id is 4134 2024-03-29T10:51:11.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9a4142b20 con 0x7ff9a4140a10 2024-03-29T10:51:11.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216278503 0 0) 0x7ff9a4144420 con 0x7ff9a409ac40 2024-03-29T10:51:11.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 monclient(hunting): my global_id is 4136 2024-03-29T10:51:11.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9a41d7280 con 0x7ff9a409ac40 2024-03-29T10:51:11.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3511735385 0 0) 0x7ff988004eb0 con 0x7ff9a41d7e70 2024-03-29T10:51:11.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9a4144420 con 0x7ff9a41d7e70 2024-03-29T10:51:11.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7ff99c001f40 con 0x7ff9a41d7e70 2024-03-29T10:51:11.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:11.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 monclient(hunting): dump: 2024-03-29T10:51:11.477 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.477 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.477 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.477 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.477 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.477 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.477 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.477 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.477 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.477 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:11.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1312149909 0 0) 0x7ff9a4142b20 con 0x7ff9a4140a10 2024-03-29T10:51:11.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff988004eb0 con 0x7ff9a4140a10 2024-03-29T10:51:11.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7ff9a00018a0 con 0x7ff9a4140a10 2024-03-29T10:51:11.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:11.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 monclient(hunting): dump: 2024-03-29T10:51:11.478 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.478 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.478 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.478 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.478 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.478 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.478 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.478 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.478 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.478 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 764209962 0 0) 0x7ff9a41d7280 con 0x7ff9a409ac40 2024-03-29T10:51:11.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9a4142b20 con 0x7ff9a409ac40 2024-03-29T10:51:11.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7ff994001770 con 0x7ff9a409ac40 2024-03-29T10:51:11.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:11.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 monclient(hunting): dump: 2024-03-29T10:51:11.479 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.479 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.479 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.479 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.479 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.479 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.479 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.479 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.479 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.479 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 121807815 0 0) 0x7ff9a4144420 con 0x7ff9a41d7e70 2024-03-29T10:51:11.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 >> v1:172.21.15.112:6790/0 conn(0x7ff9a409ac40 legacy=0x7ff9a4141690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 >> v1:172.21.15.112:6789/0 conn(0x7ff9a4140a10 legacy=0x7ff9a41d7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:51:11.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 1 monclient: found mon.b 2024-03-29T10:51:11.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:11.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:11.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.480+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a41445b0 con 0x7ff9a41d7e70 2024-03-29T10:51:11.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff99a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:51:11.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff99a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:51:11.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff99a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:11.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff9abd96640 5 monclient: authenticate success, global_id 4141 2024-03-29T10:51:11.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff9abd96640 10 monclient: _renew_subs 2024-03-29T10:51:11.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff9abd96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:11.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff9abd96640 1 -- 172.21.15.118:0/1403826602 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a4140e70 con 0x7ff9a41d7e70 2024-03-29T10:51:11.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff9abd96640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:11.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff9abd96640 10 monclient: _renew_subs 2024-03-29T10:51:11.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff9abd96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:11.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff9abd96640 1 -- 172.21.15.118:0/1403826602 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a40b1860 con 0x7ff9a41d7e70 2024-03-29T10:51:11.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff9abd96640 1 librados: init done 2024-03-29T10:51:11.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff99c004e10 con 0x7ff9a41d7e70 2024-03-29T10:51:11.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff99a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:11.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7ff99c004f80 con 0x7ff9a41d7e70 2024-03-29T10:51:11.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff99a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff99a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:11.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff99bfff640 4 set_mon_vals no callback set 2024-03-29T10:51:11.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff99a7fc640 10 monclient: dump: 2024-03-29T10:51:11.481 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.481 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.481 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.481 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.482 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.482 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.482 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.482 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.482 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.482 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff99c005470 con 0x7ff9a41d7e70 2024-03-29T10:51:11.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 566747048 0 0) 0x7ff99c006a70 con 0x7ff9a41d7e70 2024-03-29T10:51:11.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff99a7fc640 10 client.4141.objecter ms_dispatch 0x7ff9a4000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T10:51:11.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff99a7fc640 3 client.4141.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T10:51:11.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff99a7fc640 3 client.4141.objecter handle_osd_map decoding full epoch 6 2024-03-29T10:51:11.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff99a7fc640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-29T10:51:11.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff9abd96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:11.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff9abd96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:11.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff9abd96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:11.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.481+0000 7ff9abd96640 1 -- 172.21.15.118:0/1403826602 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff98c005080 con 0x7ff9a41d7e70 2024-03-29T10:51:11.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.483+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff99c006560 con 0x7ff9a41d7e70 2024-03-29T10:51:11.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.483+0000 7ff99a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:11.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.483+0000 7ff99a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:11.492 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:11.510+0000 7f0c627b8040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:51:11.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.554+0000 7ff9abd96640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "60b98350-a43a-40a3-8d6a-fa227192c87a", "id": 4}] 2024-03-29T10:51:11.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.554+0000 7ff9abd96640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "60b98350-a43a-40a3-8d6a-fa227192c87a", "id": 4}] 2024-03-29T10:51:11.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.554+0000 7ff9abd96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:11.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.554+0000 7ff9abd96640 1 -- 172.21.15.118:0/1403826602 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd new", "uuid": "60b98350-a43a-40a3-8d6a-fa227192c87a", "id": 4} v 0) v1 -- 0x7ff98c0058b0 con 0x7ff9a41d7e70 2024-03-29T10:51:11.537 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:11.556+0000 7f0c627b8040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:51:11.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.557+0000 7ff99a7fc640 1 -- 172.21.15.118:0/1403826602 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "60b98350-a43a-40a3-8d6a-fa227192c87a", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3369100289 0 1697012583) 0x7ff99c006890 con 0x7ff9a41d7e70 2024-03-29T10:51:11.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.557+0000 7ff99a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "60b98350-a43a-40a3-8d6a-fa227192c87a", "id": 4}] 2024-03-29T10:51:11.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.557+0000 7ff99a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:11.539 INFO:teuthology.orchestra.run.smithi118.stdout:4 2024-03-29T10:51:11.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.557+0000 7ff9abd96640 10 librados: watch_flush enter 2024-03-29T10:51:11.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.558+0000 7ff9abd96640 10 librados: watch_flush exit 2024-03-29T10:51:11.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.558+0000 7ff9abd96640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-29T10:51:11.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.558+0000 7ff9abd96640 10 client.4141.objecter successfully canceled tick 2024-03-29T10:51:11.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.558+0000 7ff9abd96640 10 monclient: shutdown 2024-03-29T10:51:11.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.558+0000 7ff9abd96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:11.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.558+0000 7ff9abd96640 1 -- 172.21.15.118:0/1403826602 >> v1:172.21.15.118:6789/0 conn(0x7ff9a41d7e70 legacy=0x7ff9a40b0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.558+0000 7ff9abd96640 1 -- 172.21.15.118:0/1403826602 shutdown_connections 2024-03-29T10:51:11.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.558+0000 7ff9abd96640 1 -- 172.21.15.118:0/1403826602 >> 172.21.15.118:0/1403826602 conn(0x7ff9a4066b00 msgr2=0x7ff9a413d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:11.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.558+0000 7ff9abd96640 1 -- 172.21.15.118:0/1403826602 shutdown_connections 2024-03-29T10:51:11.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.558+0000 7ff9abd96640 1 -- 172.21.15.118:0/1403826602 wait complete. 2024-03-29T10:51:11.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.558+0000 7ff9abd96640 1 librados: shutdown 2024-03-29T10:51:11.550 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d 5 2024-03-29T10:51:11.559 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:11.577+0000 7f03b67b6040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:51:11.590 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:11.608+0000 7f0c627b8040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:51:11.633 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:11.651+0000 7f03b67b6040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:51:11.636 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:11.655+0000 7f0c627b8040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:51:11.656 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.674+0000 7fddcaa4a640 10 monclient: get_monmap_and_config 2024-03-29T10:51:11.656 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.674+0000 7fddcaa4a640 10 monclient: build_initial_monmap 2024-03-29T10:51:11.656 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.674+0000 7fddcaa4a640 10 monclient: monmap: 2024-03-29T10:51:11.656 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2024-03-29T10:51:11.656 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.657 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:51:11.676057+0000 2024-03-29T10:51:11.657 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:51:11.676057+0000 2024-03-29T10:51:11.657 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:11.657 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.657 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:11.657 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:51:11.657 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:51:11.657 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.675+0000 7fddcaa4a640 1 Processor -- start 2024-03-29T10:51:11.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.675+0000 7fddcaa4a640 1 -- start start 2024-03-29T10:51:11.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.675+0000 7fddcaa4a640 10 monclient: init 2024-03-29T10:51:11.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddcaa4a640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:11.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddcaa4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:11.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddcaa4a640 10 monclient(hunting): picked mon.noname-a con 0x7fddc409ac40 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:11.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddcaa4a640 10 monclient(hunting): picked mon.noname-c con 0x7fddc409c870 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:11.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddcaa4a640 10 monclient(hunting): picked mon.noname-b con 0x7fddc40b4360 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:11.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddcaa4a640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc409dec0 con 0x7fddc409ac40 2024-03-29T10:51:11.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddcaa4a640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc409da30 con 0x7fddc409c870 2024-03-29T10:51:11.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddcaa4a640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc40b39c0 con 0x7fddc40b4360 2024-03-29T10:51:11.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddcaa4a640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:11.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc37fe640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fddc409c870 0x7fddc413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:44716/0 (socket says 172.21.15.118:44716) 2024-03-29T10:51:11.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc37fe640 1 -- 172.21.15.118:0/94496524 learned_addr learned my addr 172.21.15.118:0/94496524 (peer_addr_for_me v1:172.21.15.118:0/0) 2024-03-29T10:51:11.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddcaa4a640 10 monclient(hunting): authenticate will time out at 811.995506s 2024-03-29T10:51:11.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 1 -- 172.21.15.118:0/94496524 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129444871 0 0) 0x7fddc409da30 con 0x7fddc409c870 2024-03-29T10:51:11.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 10 monclient(hunting): my global_id is 4144 2024-03-29T10:51:11.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 1 -- 172.21.15.118:0/94496524 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddac003520 con 0x7fddc409c870 2024-03-29T10:51:11.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 1 -- 172.21.15.118:0/94496524 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618144315 0 0) 0x7fddb8001110 con 0x7fddc409ac40 2024-03-29T10:51:11.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 10 monclient(hunting): my global_id is 4137 2024-03-29T10:51:11.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 1 -- 172.21.15.118:0/94496524 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc409da30 con 0x7fddc409ac40 2024-03-29T10:51:11.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 1 -- 172.21.15.118:0/94496524 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3848049230 0 0) 0x7fddc40b39c0 con 0x7fddc40b4360 2024-03-29T10:51:11.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 10 monclient(hunting): my global_id is 4139 2024-03-29T10:51:11.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 1 -- 172.21.15.118:0/94496524 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddb8001110 con 0x7fddc40b4360 2024-03-29T10:51:11.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 1 -- 172.21.15.118:0/94496524 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1028767523 0 0) 0x7fddc409da30 con 0x7fddc409ac40 2024-03-29T10:51:11.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 1 -- 172.21.15.118:0/94496524 >> v1:172.21.15.112:6790/0 conn(0x7fddc40b4360 legacy=0x7fddc414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 1 -- 172.21.15.118:0/94496524 >> v1:172.21.15.118:6789/0 conn(0x7fddc409c870 legacy=0x7fddc413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:11.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 1 monclient: found mon.noname-a 2024-03-29T10:51:11.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:11.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T10:51:11.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 1 -- 172.21.15.118:0/94496524 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc40b5160 con 0x7fddc409ac40 2024-03-29T10:51:11.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:51:11.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:11.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:11.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 1 -- 172.21.15.118:0/94496524 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fddb8001660 con 0x7fddc409ac40 2024-03-29T10:51:11.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddcaa4a640 5 monclient: authenticate success, global_id 4137 2024-03-29T10:51:11.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddcaa4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:11.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:11.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 10 monclient: dump: 2024-03-29T10:51:11.670 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.670 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.670 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.670 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.670 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.670 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.670 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.670 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.670 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.671 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddcaa4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:11.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 1 -- 172.21.15.118:0/94496524 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddb80017c0 con 0x7fddc409ac40 2024-03-29T10:51:11.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.676+0000 7fddc2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:11.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.677+0000 7fddcaa4a640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:11.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.677+0000 7fddcaa4a640 4 set_mon_vals no callback set 2024-03-29T10:51:11.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.677+0000 7fddc2ffd640 1 -- 172.21.15.118:0/94496524 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fddb8001c10 con 0x7fddc409ac40 2024-03-29T10:51:11.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.677+0000 7fddc2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.677+0000 7fddc2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:11.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.677+0000 7fddc2ffd640 10 monclient: dump: 2024-03-29T10:51:11.671 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.671 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.671 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.671 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.672 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.672 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.672 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.672 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.672 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.672 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.677+0000 7fddcaa4a640 10 monclient: shutdown 2024-03-29T10:51:11.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.677+0000 7fddcaa4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:11.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.677+0000 7fddcaa4a640 1 -- 172.21.15.118:0/94496524 >> v1:172.21.15.112:6789/0 conn(0x7fddc409ac40 legacy=0x7fddc40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.677+0000 7fddcaa4a640 1 -- 172.21.15.118:0/94496524 shutdown_connections 2024-03-29T10:51:11.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.677+0000 7fddcaa4a640 1 -- 172.21.15.118:0/94496524 >> 172.21.15.118:0/94496524 conn(0x7fddc4066b00 msgr2=0x7fddc4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:11.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.677+0000 7fddcaa4a640 1 -- 172.21.15.118:0/94496524 shutdown_connections 2024-03-29T10:51:11.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.677+0000 7fddcaa4a640 1 -- 172.21.15.118:0/94496524 wait complete. 2024-03-29T10:51:11.673 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.678+0000 7fddcaa4a640 10 monclient: build_initial_monmap 2024-03-29T10:51:11.673 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.678+0000 7fddcaa4a640 10 monclient: monmap: 2024-03-29T10:51:11.673 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2024-03-29T10:51:11.673 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:11.673 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 0.000000 2024-03-29T10:51:11.673 INFO:teuthology.orchestra.run.smithi118.stderr:created 0.000000 2024-03-29T10:51:11.673 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:11.673 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.673 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:11.673 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:51:11.673 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:51:11.673 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.673 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.678+0000 7fddcaa4a640 1 librados: starting msgr at 2024-03-29T10:51:11.673 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.678+0000 7fddcaa4a640 1 librados: starting objecter 2024-03-29T10:51:11.673 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.678+0000 7fddcaa4a640 1 Processor -- start 2024-03-29T10:51:11.674 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.678+0000 7fddcaa4a640 1 -- start start 2024-03-29T10:51:11.674 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.678+0000 7fddcaa4a640 1 librados: setting wanted keys 2024-03-29T10:51:11.674 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.678+0000 7fddcaa4a640 1 librados: calling monclient init 2024-03-29T10:51:11.674 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.678+0000 7fddcaa4a640 10 monclient: init 2024-03-29T10:51:11.674 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddcaa4a640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:11.674 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddcaa4a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:51:11.674 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddcaa4a640 10 monclient(hunting): picked mon.noname-c con 0x7fddc409ac40 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:11.674 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddcaa4a640 10 monclient(hunting): picked mon.noname-b con 0x7fddc41d79c0 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:11.674 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddcaa4a640 10 monclient(hunting): picked mon.noname-a con 0x7fddc40b1160 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:11.674 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddcaa4a640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc41406b0 con 0x7fddc40b1160 2024-03-29T10:51:11.674 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddcaa4a640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc40b6f80 con 0x7fddc41d79c0 2024-03-29T10:51:11.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddcaa4a640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc40b7110 con 0x7fddc409ac40 2024-03-29T10:51:11.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddcaa4a640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:11.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddcaa4a640 10 monclient(hunting): authenticate will time out at 811.998456s 2024-03-29T10:51:11.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc37fe640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fddc41d79c0 0x7fddc413ad30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:44726/0 (socket says 172.21.15.118:44726) 2024-03-29T10:51:11.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc37fe640 1 -- 172.21.15.118:0/4143996527 learned_addr learned my addr 172.21.15.118:0/4143996527 (peer_addr_for_me v1:172.21.15.118:0/0) 2024-03-29T10:51:11.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 client.?.objecter ms_handle_connect 0x7fddc41d79c0 2024-03-29T10:51:11.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:11.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 client.?.objecter ms_handle_connect 0x7fddc40b1160 2024-03-29T10:51:11.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:11.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 client.?.objecter ms_handle_connect 0x7fddc409ac40 2024-03-29T10:51:11.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:11.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124612923 0 0) 0x7fddc40b6f80 con 0x7fddc41d79c0 2024-03-29T10:51:11.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 monclient(hunting): my global_id is 4147 2024-03-29T10:51:11.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddb0004eb0 con 0x7fddc41d79c0 2024-03-29T10:51:11.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3222106702 0 0) 0x7fddc41406b0 con 0x7fddc40b1160 2024-03-29T10:51:11.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 monclient(hunting): my global_id is 4140 2024-03-29T10:51:11.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc40b6f80 con 0x7fddc40b1160 2024-03-29T10:51:11.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194476302 0 0) 0x7fddc40b7110 con 0x7fddc409ac40 2024-03-29T10:51:11.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 monclient(hunting): my global_id is 4142 2024-03-29T10:51:11.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc41406b0 con 0x7fddc409ac40 2024-03-29T10:51:11.677 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 549774058 0 0) 0x7fddb0004eb0 con 0x7fddc41d79c0 2024-03-29T10:51:11.677 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddc40b7110 con 0x7fddc41d79c0 2024-03-29T10:51:11.677 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fddb4001a70 con 0x7fddc41d79c0 2024-03-29T10:51:11.677 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.677 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:11.677 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 monclient(hunting): dump: 2024-03-29T10:51:11.677 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.677 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.677 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.677 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.677 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.677 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.677 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.677 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.677 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.678 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:11.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2226028363 0 0) 0x7fddc40b6f80 con 0x7fddc40b1160 2024-03-29T10:51:11.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddb0004eb0 con 0x7fddc40b1160 2024-03-29T10:51:11.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fddbc001a70 con 0x7fddc40b1160 2024-03-29T10:51:11.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:11.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 monclient(hunting): dump: 2024-03-29T10:51:11.678 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.678 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.678 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.678 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.678 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.678 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.678 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.679 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.679 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.679 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2748060788 0 0) 0x7fddc41406b0 con 0x7fddc409ac40 2024-03-29T10:51:11.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddc40b6f80 con 0x7fddc409ac40 2024-03-29T10:51:11.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fddb80035f0 con 0x7fddc409ac40 2024-03-29T10:51:11.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.679+0000 7fddc0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:11.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc0ff9640 10 monclient(hunting): dump: 2024-03-29T10:51:11.679 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.679 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.679 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.679 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.680 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.680 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.680 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.680 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.680 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.680 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2133064476 0 0) 0x7fddc40b7110 con 0x7fddc41d79c0 2024-03-29T10:51:11.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 >> v1:172.21.15.112:6790/0 conn(0x7fddc409ac40 legacy=0x7fddc413ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 >> v1:172.21.15.112:6789/0 conn(0x7fddc40b1160 legacy=0x7fddc41e2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc0ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:51:11.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc0ff9640 1 monclient: found mon.b 2024-03-29T10:51:11.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:11.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:11.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc40b72a0 con 0x7fddc41d79c0 2024-03-29T10:51:11.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc0ff9640 10 monclient: _finish_auth 0 2024-03-29T10:51:11.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc0ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:51:11.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:11.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddcaa4a640 5 monclient: authenticate success, global_id 4147 2024-03-29T10:51:11.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddcaa4a640 10 monclient: _renew_subs 2024-03-29T10:51:11.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddcaa4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:11.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddcaa4a640 1 -- 172.21.15.118:0/4143996527 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddc413b440 con 0x7fddc41d79c0 2024-03-29T10:51:11.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddcaa4a640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:11.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddcaa4a640 10 monclient: _renew_subs 2024-03-29T10:51:11.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddcaa4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:11.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddcaa4a640 1 -- 172.21.15.118:0/4143996527 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fddc41d6dd0 con 0x7fddc41d79c0 2024-03-29T10:51:11.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddcaa4a640 1 librados: init done 2024-03-29T10:51:11.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddb40047e0 con 0x7fddc41d79c0 2024-03-29T10:51:11.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:11.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fddb4004af0 con 0x7fddc41d79c0 2024-03-29T10:51:11.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:11.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc27fc640 4 set_mon_vals no callback set 2024-03-29T10:51:11.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc0ff9640 10 monclient: dump: 2024-03-29T10:51:11.682 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.682 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.682 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.682 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.682 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.682 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.682 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.682 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.682 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.683 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fddb4005060 con 0x7fddc41d79c0 2024-03-29T10:51:11.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddcaa4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:11.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 1018823056 0 0) 0x7fddb40068c0 con 0x7fddc41d79c0 2024-03-29T10:51:11.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc0ff9640 10 client.4147.objecter ms_dispatch 0x7fddc4000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T10:51:11.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc0ff9640 3 client.4147.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T10:51:11.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.680+0000 7fddc0ff9640 3 client.4147.objecter handle_osd_map decoding full epoch 7 2024-03-29T10:51:11.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.681+0000 7fddc0ff9640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-29T10:51:11.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.681+0000 7fddcaa4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:11.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.681+0000 7fddcaa4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:11.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.681+0000 7fddcaa4a640 1 -- 172.21.15.118:0/4143996527 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdda4005080 con 0x7fddc41d79c0 2024-03-29T10:51:11.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.682+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fddb40063f0 con 0x7fddc41d79c0 2024-03-29T10:51:11.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.682+0000 7fddc0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:11.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.682+0000 7fddc0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:11.684 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:11.700+0000 7f03b67b6040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:51:11.736 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:11.754+0000 7f03b67b6040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:51:11.736 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.754+0000 7fddcaa4a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d", "id": 5}] 2024-03-29T10:51:11.736 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.754+0000 7fddcaa4a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d", "id": 5}] 2024-03-29T10:51:11.737 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.754+0000 7fddcaa4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:11.737 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.754+0000 7fddcaa4a640 1 -- 172.21.15.118:0/4143996527 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd new", "uuid": "bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d", "id": 5} v 0) v1 -- 0x7fdda40058b0 con 0x7fddc41d79c0 2024-03-29T10:51:11.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.757+0000 7fddc0ff9640 1 -- 172.21.15.118:0/4143996527 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3678352781 0 1988412176) 0x7fddb4006660 con 0x7fddc41d79c0 2024-03-29T10:51:11.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.757+0000 7fddc0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d", "id": 5}] 2024-03-29T10:51:11.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.757+0000 7fddc0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:11.740 INFO:teuthology.orchestra.run.smithi118.stdout:5 2024-03-29T10:51:11.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.758+0000 7fddcaa4a640 10 librados: watch_flush enter 2024-03-29T10:51:11.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.758+0000 7fddcaa4a640 10 librados: watch_flush exit 2024-03-29T10:51:11.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.758+0000 7fddcaa4a640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-29T10:51:11.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.758+0000 7fddcaa4a640 10 client.4147.objecter successfully canceled tick 2024-03-29T10:51:11.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.758+0000 7fddcaa4a640 10 monclient: shutdown 2024-03-29T10:51:11.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.758+0000 7fddcaa4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:11.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.758+0000 7fddcaa4a640 1 -- 172.21.15.118:0/4143996527 >> v1:172.21.15.118:6789/0 conn(0x7fddc41d79c0 legacy=0x7fddc413ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.758+0000 7fddcaa4a640 1 -- 172.21.15.118:0/4143996527 shutdown_connections 2024-03-29T10:51:11.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.758+0000 7fddcaa4a640 1 -- 172.21.15.118:0/4143996527 >> 172.21.15.118:0/4143996527 conn(0x7fddc4066b00 msgr2=0x7fddc413bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:11.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.758+0000 7fddcaa4a640 1 -- 172.21.15.118:0/4143996527 shutdown_connections 2024-03-29T10:51:11.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.758+0000 7fddcaa4a640 1 -- 172.21.15.118:0/4143996527 wait complete. 2024-03-29T10:51:11.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.758+0000 7fddcaa4a640 1 librados: shutdown 2024-03-29T10:51:11.750 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new ffd83441-b00f-42ba-ba42-69f1fadb69eb 6 2024-03-29T10:51:11.787 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:11.805+0000 7f03b67b6040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:51:11.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.867+0000 7fc931aae640 10 monclient: get_monmap_and_config 2024-03-29T10:51:11.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.867+0000 7fc931aae640 10 monclient: build_initial_monmap 2024-03-29T10:51:11.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.867+0000 7fc931aae640 10 monclient: monmap: 2024-03-29T10:51:11.849 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2024-03-29T10:51:11.849 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.849 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:51:11.868414+0000 2024-03-29T10:51:11.849 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:51:11.868414+0000 2024-03-29T10:51:11.849 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:11.849 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.849 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:11.849 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:51:11.850 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:51:11.850 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.859 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.867+0000 7fc931aae640 1 Processor -- start 2024-03-29T10:51:11.859 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.867+0000 7fc931aae640 1 -- start start 2024-03-29T10:51:11.859 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.867+0000 7fc931aae640 10 monclient: init 2024-03-29T10:51:11.859 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc931aae640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:11.859 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc931aae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:11.859 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc931aae640 10 monclient(hunting): picked mon.noname-a con 0x7fc92c09ac40 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:11.859 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc931aae640 10 monclient(hunting): picked mon.noname-c con 0x7fc92c09c870 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:11.859 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc931aae640 10 monclient(hunting): picked mon.noname-b con 0x7fc92c0b4360 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:11.859 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc931aae640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc92c09dec0 con 0x7fc92c09ac40 2024-03-29T10:51:11.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc931aae640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc92c09da30 con 0x7fc92c09c870 2024-03-29T10:51:11.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc931aae640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc92c0b39c0 con 0x7fc92c0b4360 2024-03-29T10:51:11.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc931aae640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:11.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc931aae640 10 monclient(hunting): authenticate will time out at 812.187652s 2024-03-29T10:51:11.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc92affd640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fc92c09c870 0x7fc92c13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:44732/0 (socket says 172.21.15.118:44732) 2024-03-29T10:51:11.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc92affd640 1 -- 172.21.15.118:0/1805538424 learned_addr learned my addr 172.21.15.118:0/1805538424 (peer_addr_for_me v1:172.21.15.118:0/0) 2024-03-29T10:51:11.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc92a7fc640 1 -- 172.21.15.118:0/1805538424 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2463728585 0 0) 0x7fc92c09da30 con 0x7fc92c09c870 2024-03-29T10:51:11.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc92a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc92a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc92a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc92a7fc640 10 monclient(hunting): my global_id is 4150 2024-03-29T10:51:11.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc92a7fc640 1 -- 172.21.15.118:0/1805538424 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc914003520 con 0x7fc92c09c870 2024-03-29T10:51:11.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc92a7fc640 1 -- 172.21.15.118:0/1805538424 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677122084 0 0) 0x7fc92c0b39c0 con 0x7fc92c0b4360 2024-03-29T10:51:11.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc92a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc92a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc92a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc92a7fc640 10 monclient(hunting): my global_id is 4145 2024-03-29T10:51:11.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc92a7fc640 1 -- 172.21.15.118:0/1805538424 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc92c09da30 con 0x7fc92c0b4360 2024-03-29T10:51:11.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc92a7fc640 1 -- 172.21.15.118:0/1805538424 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99311986 0 0) 0x7fc920001110 con 0x7fc92c09ac40 2024-03-29T10:51:11.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc92a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc92a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc92a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.868+0000 7fc92a7fc640 10 monclient(hunting): my global_id is 4143 2024-03-29T10:51:11.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc92a7fc640 1 -- 172.21.15.118:0/1805538424 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc92c0b39c0 con 0x7fc92c09ac40 2024-03-29T10:51:11.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc92a7fc640 1 -- 172.21.15.118:0/1805538424 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2076079979 0 0) 0x7fc914003520 con 0x7fc92c09c870 2024-03-29T10:51:11.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc92a7fc640 1 -- 172.21.15.118:0/1805538424 >> v1:172.21.15.112:6790/0 conn(0x7fc92c0b4360 legacy=0x7fc92c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc92a7fc640 1 -- 172.21.15.118:0/1805538424 >> v1:172.21.15.112:6789/0 conn(0x7fc92c09ac40 legacy=0x7fc92c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc92a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:51:11.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc92a7fc640 1 monclient: found mon.noname-c 2024-03-29T10:51:11.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc92a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:11.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc92a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T10:51:11.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc92a7fc640 1 -- 172.21.15.118:0/1805538424 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc92c0b5160 con 0x7fc92c09c870 2024-03-29T10:51:11.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc92a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:51:11.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc92a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:51:11.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc92a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:11.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc92a7fc640 1 -- 172.21.15.118:0/1805538424 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fc91c001610 con 0x7fc92c09c870 2024-03-29T10:51:11.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc92a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc92a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:11.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc92a7fc640 10 monclient: dump: 2024-03-29T10:51:11.862 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.862 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.863 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc931aae640 5 monclient: authenticate success, global_id 4150 2024-03-29T10:51:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc931aae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc92a7fc640 1 -- 172.21.15.118:0/1805538424 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc91c001770 con 0x7fc92c09c870 2024-03-29T10:51:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc92a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc931aae640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc92a7fc640 1 -- 172.21.15.118:0/1805538424 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fc91c001bc0 con 0x7fc92c09c870 2024-03-29T10:51:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc92a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc931aae640 4 set_mon_vals no callback set 2024-03-29T10:51:11.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc92a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:11.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc92a7fc640 10 monclient: dump: 2024-03-29T10:51:11.864 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.864 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.864 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.864 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.864 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.864 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.864 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.864 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.864 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.864 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc931aae640 10 monclient: shutdown 2024-03-29T10:51:11.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc931aae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:11.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc931aae640 1 -- 172.21.15.118:0/1805538424 >> v1:172.21.15.118:6789/0 conn(0x7fc92c09c870 legacy=0x7fc92c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc931aae640 1 -- 172.21.15.118:0/1805538424 shutdown_connections 2024-03-29T10:51:11.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc931aae640 1 -- 172.21.15.118:0/1805538424 >> 172.21.15.118:0/1805538424 conn(0x7fc92c066b00 msgr2=0x7fc92c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:11.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc931aae640 1 -- 172.21.15.118:0/1805538424 shutdown_connections 2024-03-29T10:51:11.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.869+0000 7fc931aae640 1 -- 172.21.15.118:0/1805538424 wait complete. 2024-03-29T10:51:11.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.870+0000 7fc931aae640 10 monclient: build_initial_monmap 2024-03-29T10:51:11.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.870+0000 7fc931aae640 10 monclient: monmap: 2024-03-29T10:51:11.865 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2024-03-29T10:51:11.865 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:11.865 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 0.000000 2024-03-29T10:51:11.865 INFO:teuthology.orchestra.run.smithi118.stderr:created 0.000000 2024-03-29T10:51:11.865 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:11.865 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.866 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:11.866 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:51:11.866 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:51:11.866 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.870+0000 7fc931aae640 1 librados: starting msgr at 2024-03-29T10:51:11.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.870+0000 7fc931aae640 1 librados: starting objecter 2024-03-29T10:51:11.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.870+0000 7fc931aae640 1 Processor -- start 2024-03-29T10:51:11.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.870+0000 7fc931aae640 1 -- start start 2024-03-29T10:51:11.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.870+0000 7fc931aae640 1 librados: setting wanted keys 2024-03-29T10:51:11.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.870+0000 7fc931aae640 1 librados: calling monclient init 2024-03-29T10:51:11.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.870+0000 7fc931aae640 10 monclient: init 2024-03-29T10:51:11.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc931aae640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:11.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc931aae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:51:11.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc931aae640 10 monclient(hunting): picked mon.noname-b con 0x7fc92c09ac40 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:11.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc931aae640 10 monclient(hunting): picked mon.noname-a con 0x7fc92c0afbd0 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:11.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc931aae640 10 monclient(hunting): picked mon.noname-c con 0x7fc92c1db2f0 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:11.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc931aae640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc92c0b0060 con 0x7fc92c0afbd0 2024-03-29T10:51:11.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc931aae640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc92c11f8d0 con 0x7fc92c09ac40 2024-03-29T10:51:11.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc931aae640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc92c0b0d20 con 0x7fc92c1db2f0 2024-03-29T10:51:11.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc931aae640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:11.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc931aae640 10 monclient(hunting): authenticate will time out at 812.190633s 2024-03-29T10:51:11.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc92b7fe640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fc92c09ac40 0x7fc92c0b0610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:44734/0 (socket says 172.21.15.118:44734) 2024-03-29T10:51:11.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc92b7fe640 1 -- 172.21.15.118:0/3820724161 learned_addr learned my addr 172.21.15.118:0/3820724161 (peer_addr_for_me v1:172.21.15.118:0/0) 2024-03-29T10:51:11.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 10 client.?.objecter ms_handle_connect 0x7fc92c09ac40 2024-03-29T10:51:11.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:11.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 10 client.?.objecter ms_handle_connect 0x7fc92c0afbd0 2024-03-29T10:51:11.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:11.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 10 client.?.objecter ms_handle_connect 0x7fc92c1db2f0 2024-03-29T10:51:11.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:11.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297091114 0 0) 0x7fc92c11f8d0 con 0x7fc92c09ac40 2024-03-29T10:51:11.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 10 monclient(hunting): my global_id is 4153 2024-03-29T10:51:11.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc918004eb0 con 0x7fc92c09ac40 2024-03-29T10:51:11.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578181829 0 0) 0x7fc92c0b0060 con 0x7fc92c0afbd0 2024-03-29T10:51:11.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 10 monclient(hunting): my global_id is 4146 2024-03-29T10:51:11.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc92c11f8d0 con 0x7fc92c0afbd0 2024-03-29T10:51:11.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378846289 0 0) 0x7fc92c0b0d20 con 0x7fc92c1db2f0 2024-03-29T10:51:11.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:11.869 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:11.869 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:11.869 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 10 monclient(hunting): my global_id is 4148 2024-03-29T10:51:11.869 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc92c0b0060 con 0x7fc92c1db2f0 2024-03-29T10:51:11.869 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.871+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4067342524 0 0) 0x7fc918004eb0 con 0x7fc92c09ac40 2024-03-29T10:51:11.869 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc92c0b0d20 con 0x7fc92c09ac40 2024-03-29T10:51:11.869 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fc9200018e0 con 0x7fc92c09ac40 2024-03-29T10:51:11.869 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.869 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:11.869 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 10 monclient(hunting): dump: 2024-03-29T10:51:11.869 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.869 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.869 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.869 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.869 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.870 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3640102084 0 0) 0x7fc92c11f8d0 con 0x7fc92c0afbd0 2024-03-29T10:51:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc918004eb0 con 0x7fc92c0afbd0 2024-03-29T10:51:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fc91c001a80 con 0x7fc92c0afbd0 2024-03-29T10:51:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 10 monclient(hunting): dump: 2024-03-29T10:51:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.871 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4062392545 0 0) 0x7fc92c0b0060 con 0x7fc92c1db2f0 2024-03-29T10:51:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc92c11f8d0 con 0x7fc92c1db2f0 2024-03-29T10:51:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fc924001bd0 con 0x7fc92c1db2f0 2024-03-29T10:51:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 10 monclient(hunting): dump: 2024-03-29T10:51:11.872 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.872 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.872 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.872 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.872 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.872 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.872 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.872 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.872 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.872 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2040796487 0 0) 0x7fc92c0b0d20 con 0x7fc92c09ac40 2024-03-29T10:51:11.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 >> v1:172.21.15.112:6790/0 conn(0x7fc92c1db2f0 legacy=0x7fc92c1e1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 >> v1:172.21.15.112:6789/0 conn(0x7fc92c0afbd0 legacy=0x7fc92c1dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 10 monclient: _finish_hunting 0 2024-03-29T10:51:11.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 1 monclient: found mon.b 2024-03-29T10:51:11.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:11.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:11.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc92c0acf60 con 0x7fc92c09ac40 2024-03-29T10:51:11.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 10 monclient: _finish_auth 0 2024-03-29T10:51:11.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 10 monclient: _check_auth_tickets 2024-03-29T10:51:11.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:11.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc931aae640 5 monclient: authenticate success, global_id 4153 2024-03-29T10:51:11.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc931aae640 10 monclient: _renew_subs 2024-03-29T10:51:11.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc931aae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:11.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc931aae640 1 -- 172.21.15.118:0/3820724161 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc92c1e3390 con 0x7fc92c09ac40 2024-03-29T10:51:11.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc931aae640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:11.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc931aae640 10 monclient: _renew_subs 2024-03-29T10:51:11.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc931aae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:11.874 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc931aae640 1 -- 172.21.15.118:0/3820724161 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc92c1e3700 con 0x7fc92c09ac40 2024-03-29T10:51:11.874 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc931aae640 1 librados: init done 2024-03-29T10:51:11.874 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc920003610 con 0x7fc92c09ac40 2024-03-29T10:51:11.874 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:11.874 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc929ffb640 4 set_mon_vals no callback set 2024-03-29T10:51:11.874 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fc920003a60 con 0x7fc92c09ac40 2024-03-29T10:51:11.874 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:11.874 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:11.874 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 10 monclient: dump: 2024-03-29T10:51:11.874 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:11.874 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:11.874 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.874 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:11.874 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:11.875 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:11.875 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:11.875 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:11.875 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:11.875 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:11.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc920003f50 con 0x7fc92c09ac40 2024-03-29T10:51:11.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 4212615468 0 0) 0x7fc920006700 con 0x7fc92c09ac40 2024-03-29T10:51:11.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 10 client.4153.objecter ms_dispatch 0x7fc92c000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T10:51:11.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 3 client.4153.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T10:51:11.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 3 client.4153.objecter handle_osd_map decoding full epoch 8 2024-03-29T10:51:11.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.872+0000 7fc913fff640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-29T10:51:11.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.873+0000 7fc931aae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:11.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.873+0000 7fc931aae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:11.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.873+0000 7fc931aae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:11.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.873+0000 7fc931aae640 1 -- 172.21.15.118:0/3820724161 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc90c005080 con 0x7fc92c09ac40 2024-03-29T10:51:11.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.875+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc9200057c0 con 0x7fc92c09ac40 2024-03-29T10:51:11.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.875+0000 7fc913fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:11.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.875+0000 7fc913fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:11.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.946+0000 7fc931aae640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ffd83441-b00f-42ba-ba42-69f1fadb69eb", "id": 6}] 2024-03-29T10:51:11.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.946+0000 7fc931aae640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ffd83441-b00f-42ba-ba42-69f1fadb69eb", "id": 6}] 2024-03-29T10:51:11.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.946+0000 7fc931aae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:11.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.946+0000 7fc931aae640 1 -- 172.21.15.118:0/3820724161 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ffd83441-b00f-42ba-ba42-69f1fadb69eb", "id": 6} v 0) v1 -- 0x7fc90c0058b0 con 0x7fc92c09ac40 2024-03-29T10:51:11.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.949+0000 7fc913fff640 1 -- 172.21.15.118:0/3820724161 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ffd83441-b00f-42ba-ba42-69f1fadb69eb", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2248626051 0 1113811849) 0x7fc920005960 con 0x7fc92c09ac40 2024-03-29T10:51:11.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.949+0000 7fc913fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ffd83441-b00f-42ba-ba42-69f1fadb69eb", "id": 6}] 2024-03-29T10:51:11.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.949+0000 7fc913fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:11.931 INFO:teuthology.orchestra.run.smithi118.stdout:6 2024-03-29T10:51:11.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.950+0000 7fc931aae640 10 librados: watch_flush enter 2024-03-29T10:51:11.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.950+0000 7fc931aae640 10 librados: watch_flush exit 2024-03-29T10:51:11.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.950+0000 7fc931aae640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-29T10:51:11.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.950+0000 7fc931aae640 10 client.4153.objecter successfully canceled tick 2024-03-29T10:51:11.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.950+0000 7fc931aae640 10 monclient: shutdown 2024-03-29T10:51:11.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.950+0000 7fc931aae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:11.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.950+0000 7fc931aae640 1 -- 172.21.15.118:0/3820724161 >> v1:172.21.15.118:6789/0 conn(0x7fc92c09ac40 legacy=0x7fc92c0b0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.950+0000 7fc931aae640 1 -- 172.21.15.118:0/3820724161 shutdown_connections 2024-03-29T10:51:11.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.950+0000 7fc931aae640 1 -- 172.21.15.118:0/3820724161 >> 172.21.15.118:0/3820724161 conn(0x7fc92c066b00 msgr2=0x7fc92c13d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:11.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.950+0000 7fc931aae640 1 -- 172.21.15.118:0/3820724161 shutdown_connections 2024-03-29T10:51:11.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.950+0000 7fc931aae640 1 -- 172.21.15.118:0/3820724161 wait complete. 2024-03-29T10:51:11.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:11.950+0000 7fc931aae640 1 librados: shutdown 2024-03-29T10:51:11.942 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new 4f5b800e-e099-41e8-8fdc-4a5ad8828199 7 2024-03-29T10:51:12.017 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:12.035+0000 7f0c627b8040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:51:12.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.100+0000 7fc7266c8640 10 monclient: get_monmap_and_config 2024-03-29T10:51:12.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.100+0000 7fc7266c8640 10 monclient: build_initial_monmap 2024-03-29T10:51:12.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.100+0000 7fc7266c8640 10 monclient: monmap: 2024-03-29T10:51:12.083 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2024-03-29T10:51:12.083 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:12.083 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:51:12.102125+0000 2024-03-29T10:51:12.083 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:51:12.102125+0000 2024-03-29T10:51:12.083 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:12.083 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:12.083 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:12.083 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:51:12.083 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:51:12.083 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:12.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.101+0000 7fc7266c8640 1 Processor -- start 2024-03-29T10:51:12.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.101+0000 7fc7266c8640 1 -- start start 2024-03-29T10:51:12.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.101+0000 7fc7266c8640 10 monclient: init 2024-03-29T10:51:12.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc7266c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:12.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc7266c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:12.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc7266c8640 10 monclient(hunting): picked mon.noname-b con 0x7fc72009ac40 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:12.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc7266c8640 10 monclient(hunting): picked mon.noname-a con 0x7fc72009c870 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:12.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc7266c8640 10 monclient(hunting): picked mon.noname-c con 0x7fc7200b4360 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:12.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc7266c8640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc72009dec0 con 0x7fc72009c870 2024-03-29T10:51:12.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc7266c8640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc72009da30 con 0x7fc7200b4360 2024-03-29T10:51:12.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc7266c8640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7200b39c0 con 0x7fc72009ac40 2024-03-29T10:51:12.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc7266c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:12.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc7266c8640 10 monclient(hunting): authenticate will time out at 812.421476s 2024-03-29T10:51:12.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc724c65640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fc7200b4360 0x7fc72014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:44748/0 (socket says 172.21.15.118:44748) 2024-03-29T10:51:12.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc724c65640 1 -- 172.21.15.118:0/3576865068 learned_addr learned my addr 172.21.15.118:0/3576865068 (peer_addr_for_me v1:172.21.15.118:0/0) 2024-03-29T10:51:12.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 1 -- 172.21.15.118:0/3576865068 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192879585 0 0) 0x7fc72009da30 con 0x7fc7200b4360 2024-03-29T10:51:12.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:12.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:12.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:12.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 10 monclient(hunting): my global_id is 4156 2024-03-29T10:51:12.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 1 -- 172.21.15.118:0/3576865068 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc708003520 con 0x7fc7200b4360 2024-03-29T10:51:12.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 1 -- 172.21.15.118:0/3576865068 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818052377 0 0) 0x7fc7200b39c0 con 0x7fc72009ac40 2024-03-29T10:51:12.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:12.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:12.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:12.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 10 monclient(hunting): my global_id is 4151 2024-03-29T10:51:12.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 1 -- 172.21.15.118:0/3576865068 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc72009da30 con 0x7fc72009ac40 2024-03-29T10:51:12.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 1 -- 172.21.15.118:0/3576865068 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3992828088 0 0) 0x7fc710001110 con 0x7fc72009c870 2024-03-29T10:51:12.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:12.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:12.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:12.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 10 monclient(hunting): my global_id is 4149 2024-03-29T10:51:12.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 1 -- 172.21.15.118:0/3576865068 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7200b39c0 con 0x7fc72009c870 2024-03-29T10:51:12.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 1 -- 172.21.15.118:0/3576865068 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4189765747 0 0) 0x7fc708003520 con 0x7fc7200b4360 2024-03-29T10:51:12.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 1 -- 172.21.15.118:0/3576865068 >> v1:172.21.15.112:6790/0 conn(0x7fc72009ac40 legacy=0x7fc7200b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:12.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 1 -- 172.21.15.118:0/3576865068 >> v1:172.21.15.112:6789/0 conn(0x7fc72009c870 legacy=0x7fc72013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:12.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:12.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 1 monclient: found mon.noname-c 2024-03-29T10:51:12.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:12.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T10:51:12.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 1 -- 172.21.15.118:0/3576865068 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7200b5160 con 0x7fc7200b4360 2024-03-29T10:51:12.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 10 monclient: _finish_auth 0 2024-03-29T10:51:12.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:12.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:12.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc7266c8640 5 monclient: authenticate success, global_id 4156 2024-03-29T10:51:12.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 1 -- 172.21.15.118:0/3576865068 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fc718001990 con 0x7fc7200b4360 2024-03-29T10:51:12.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc7266c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:12.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:12.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:12.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 10 monclient: dump: 2024-03-29T10:51:12.096 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:12.096 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:12.096 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:12.096 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:12.096 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:12.096 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:12.096 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:12.096 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:12.096 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:12.096 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:12.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc7266c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:12.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 1 -- 172.21.15.118:0/3576865068 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc718001af0 con 0x7fc7200b4360 2024-03-29T10:51:12.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:12.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 1 -- 172.21.15.118:0/3576865068 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fc718001f40 con 0x7fc7200b4360 2024-03-29T10:51:12.097 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc7266c8640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:12.097 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc7266c8640 4 set_mon_vals no callback set 2024-03-29T10:51:12.097 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:12.097 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.102+0000 7fc71effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:12.097 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.103+0000 7fc71effd640 10 monclient: dump: 2024-03-29T10:51:12.097 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:12.097 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:12.097 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:12.097 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:12.097 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:12.097 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:12.097 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:12.097 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:12.097 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:12.097 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:12.098 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.103+0000 7fc7266c8640 10 monclient: shutdown 2024-03-29T10:51:12.098 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.103+0000 7fc7266c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:12.098 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.103+0000 7fc7266c8640 1 -- 172.21.15.118:0/3576865068 >> v1:172.21.15.118:6789/0 conn(0x7fc7200b4360 legacy=0x7fc72014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:12.098 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.103+0000 7fc7266c8640 1 -- 172.21.15.118:0/3576865068 shutdown_connections 2024-03-29T10:51:12.098 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.103+0000 7fc7266c8640 1 -- 172.21.15.118:0/3576865068 >> 172.21.15.118:0/3576865068 conn(0x7fc720066b00 msgr2=0x7fc720066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:12.098 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.103+0000 7fc7266c8640 1 -- 172.21.15.118:0/3576865068 shutdown_connections 2024-03-29T10:51:12.098 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.103+0000 7fc7266c8640 1 -- 172.21.15.118:0/3576865068 wait complete. 2024-03-29T10:51:12.098 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.104+0000 7fc7266c8640 10 monclient: build_initial_monmap 2024-03-29T10:51:12.098 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.104+0000 7fc7266c8640 10 monclient: monmap: 2024-03-29T10:51:12.098 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2024-03-29T10:51:12.098 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:12.098 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 0.000000 2024-03-29T10:51:12.098 INFO:teuthology.orchestra.run.smithi118.stderr:created 0.000000 2024-03-29T10:51:12.099 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:12.099 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:12.099 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:12.099 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:51:12.099 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:51:12.099 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:12.099 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.104+0000 7fc7266c8640 1 librados: starting msgr at 2024-03-29T10:51:12.099 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.104+0000 7fc7266c8640 1 librados: starting objecter 2024-03-29T10:51:12.099 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.104+0000 7fc7266c8640 1 Processor -- start 2024-03-29T10:51:12.099 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.104+0000 7fc7266c8640 1 -- start start 2024-03-29T10:51:12.099 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.104+0000 7fc7266c8640 1 librados: setting wanted keys 2024-03-29T10:51:12.099 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.104+0000 7fc7266c8640 1 librados: calling monclient init 2024-03-29T10:51:12.099 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.104+0000 7fc7266c8640 10 monclient: init 2024-03-29T10:51:12.099 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc7266c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:12.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc7266c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:12.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc7266c8640 10 monclient(hunting): picked mon.noname-c con 0x7fc72009ac40 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:12.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc7266c8640 10 monclient(hunting): picked mon.noname-a con 0x7fc7200aecc0 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:12.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc7266c8640 10 monclient(hunting): picked mon.noname-b con 0x7fc7200b0540 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:12.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc7266c8640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7200b6e10 con 0x7fc7200aecc0 2024-03-29T10:51:12.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc7266c8640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc720134900 con 0x7fc7200b0540 2024-03-29T10:51:12.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc7266c8640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7200afd60 con 0x7fc72009ac40 2024-03-29T10:51:12.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc7266c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:12.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc7266c8640 10 monclient(hunting): authenticate will time out at 812.424476s 2024-03-29T10:51:12.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc724c65640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fc7200b0540 0x7fc7201341f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:44764/0 (socket says 172.21.15.118:44764) 2024-03-29T10:51:12.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc724c65640 1 -- 172.21.15.118:0/1284723971 learned_addr learned my addr 172.21.15.118:0/1284723971 (peer_addr_for_me v1:172.21.15.118:0/0) 2024-03-29T10:51:12.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 client.?.objecter ms_handle_connect 0x7fc7200b0540 2024-03-29T10:51:12.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:12.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 client.?.objecter ms_handle_connect 0x7fc72009ac40 2024-03-29T10:51:12.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:12.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 client.?.objecter ms_handle_connect 0x7fc7200aecc0 2024-03-29T10:51:12.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:12.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552953553 0 0) 0x7fc720134900 con 0x7fc7200b0540 2024-03-29T10:51:12.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:12.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:12.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:12.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 monclient(hunting): my global_id is 4159 2024-03-29T10:51:12.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc70c004eb0 con 0x7fc7200b0540 2024-03-29T10:51:12.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3098188037 0 0) 0x7fc7200afd60 con 0x7fc72009ac40 2024-03-29T10:51:12.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:12.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:12.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 monclient(hunting): my global_id is 4154 2024-03-29T10:51:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc720134900 con 0x7fc72009ac40 2024-03-29T10:51:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3987946375 0 0) 0x7fc7200b6e10 con 0x7fc7200aecc0 2024-03-29T10:51:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 monclient(hunting): my global_id is 4152 2024-03-29T10:51:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7200afd60 con 0x7fc7200aecc0 2024-03-29T10:51:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1802477164 0 0) 0x7fc70c004eb0 con 0x7fc7200b0540 2024-03-29T10:51:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7200b6e10 con 0x7fc7200b0540 2024-03-29T10:51:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fc718001ec0 con 0x7fc7200b0540 2024-03-29T10:51:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 monclient(hunting): dump: 2024-03-29T10:51:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:12.103 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:12.103 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:12.103 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:12.103 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:12.103 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:12.103 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:12.103 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:12.103 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:12.103 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:12.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:12.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1981899512 0 0) 0x7fc7200afd60 con 0x7fc7200aecc0 2024-03-29T10:51:12.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc70c004eb0 con 0x7fc7200aecc0 2024-03-29T10:51:12.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1850415114 0 0) 0x7fc720134900 con 0x7fc72009ac40 2024-03-29T10:51:12.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7200afd60 con 0x7fc72009ac40 2024-03-29T10:51:12.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fc710001860 con 0x7fc7200aecc0 2024-03-29T10:51:12.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:12.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:12.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 monclient(hunting): dump: 2024-03-29T10:51:12.104 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:12.104 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:12.104 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:12.104 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:12.104 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:12.104 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:12.104 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:12.104 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:12.104 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:12.104 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:12.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fc7140018f0 con 0x7fc72009ac40 2024-03-29T10:51:12.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:12.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:12.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 10 monclient(hunting): dump: 2024-03-29T10:51:12.105 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:12.105 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:12.105 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:12.105 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:12.105 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:12.105 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:12.105 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:12.105 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:12.105 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:12.105 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:12.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.105+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1500506797 0 0) 0x7fc7200b6e10 con 0x7fc7200b0540 2024-03-29T10:51:12.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 >> v1:172.21.15.112:6790/0 conn(0x7fc72009ac40 legacy=0x7fc7200b1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:12.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 >> v1:172.21.15.112:6789/0 conn(0x7fc7200aecc0 legacy=0x7fc720133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:12.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc71cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:51:12.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc71cff9640 1 monclient: found mon.b 2024-03-29T10:51:12.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc71cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:12.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc71cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:12.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7200ae680 con 0x7fc7200b0540 2024-03-29T10:51:12.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc71cff9640 10 monclient: _finish_auth 0 2024-03-29T10:51:12.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc71cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:51:12.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc71cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:12.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc7266c8640 5 monclient: authenticate success, global_id 4159 2024-03-29T10:51:12.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc7266c8640 10 monclient: _renew_subs 2024-03-29T10:51:12.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc7266c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:12.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc7266c8640 1 -- 172.21.15.118:0/1284723971 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7200ae870 con 0x7fc7200b0540 2024-03-29T10:51:12.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc7266c8640 10 client.4159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:12.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc7266c8640 10 monclient: _renew_subs 2024-03-29T10:51:12.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc7266c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:12.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc7266c8640 1 -- 172.21.15.118:0/1284723971 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7200b1920 con 0x7fc7200b0540 2024-03-29T10:51:12.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc7266c8640 1 librados: init done 2024-03-29T10:51:12.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7180020e0 con 0x7fc7200b0540 2024-03-29T10:51:12.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc71cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:12.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fc718004ff0 con 0x7fc7200b0540 2024-03-29T10:51:12.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc71cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:12.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc71e7fc640 4 set_mon_vals no callback set 2024-03-29T10:51:12.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc71cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:12.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc71cff9640 10 monclient: dump: 2024-03-29T10:51:12.107 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2024-03-29T10:51:12.107 INFO:teuthology.orchestra.run.smithi118.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:12.107 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:12.107 INFO:teuthology.orchestra.run.smithi118.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:12.108 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:12.108 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2024-03-29T10:51:12.108 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:12.108 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:12.108 INFO:teuthology.orchestra.run.smithi118.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:12.108 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T10:51:12.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc718005560 con 0x7fc7200b0540 2024-03-29T10:51:12.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2353086082 0 0) 0x7fc718005e70 con 0x7fc7200b0540 2024-03-29T10:51:12.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc71cff9640 10 client.4159.objecter ms_dispatch 0x7fc720000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T10:51:12.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc71cff9640 3 client.4159.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T10:51:12.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc71cff9640 3 client.4159.objecter handle_osd_map decoding full epoch 9 2024-03-29T10:51:12.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc71cff9640 20 client.4159.objecter dump_active .. 0 homeless 2024-03-29T10:51:12.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.106+0000 7fc7266c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:12.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.107+0000 7fc7266c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:12.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.107+0000 7fc7266c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:12.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.107+0000 7fc7266c8640 1 -- 172.21.15.118:0/1284723971 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc700005080 con 0x7fc7200b0540 2024-03-29T10:51:12.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.108+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc718006150 con 0x7fc7200b0540 2024-03-29T10:51:12.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.108+0000 7fc71cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:12.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.108+0000 7fc71cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:12.153 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:12.171+0000 7f0c627b8040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:51:12.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.181+0000 7fc7266c8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4f5b800e-e099-41e8-8fdc-4a5ad8828199", "id": 7}] 2024-03-29T10:51:12.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.181+0000 7fc7266c8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4f5b800e-e099-41e8-8fdc-4a5ad8828199", "id": 7}] 2024-03-29T10:51:12.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.181+0000 7fc7266c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:12.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.181+0000 7fc7266c8640 1 -- 172.21.15.118:0/1284723971 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd new", "uuid": "4f5b800e-e099-41e8-8fdc-4a5ad8828199", "id": 7} v 0) v1 -- 0x7fc7000058b0 con 0x7fc7200b0540 2024-03-29T10:51:12.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.184+0000 7fc71cff9640 1 -- 172.21.15.118:0/1284723971 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4f5b800e-e099-41e8-8fdc-4a5ad8828199", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 2837758089 0 1371665406) 0x7fc7180058e0 con 0x7fc7200b0540 2024-03-29T10:51:12.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.184+0000 7fc71cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4f5b800e-e099-41e8-8fdc-4a5ad8828199", "id": 7}] 2024-03-29T10:51:12.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.184+0000 7fc71cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:12.166 INFO:teuthology.orchestra.run.smithi118.stdout:7 2024-03-29T10:51:12.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.185+0000 7fc7266c8640 10 librados: watch_flush enter 2024-03-29T10:51:12.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.185+0000 7fc7266c8640 10 librados: watch_flush exit 2024-03-29T10:51:12.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.185+0000 7fc7266c8640 20 client.4159.objecter shutdown clearing up homeless session... 2024-03-29T10:51:12.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.185+0000 7fc7266c8640 10 client.4159.objecter successfully canceled tick 2024-03-29T10:51:12.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.185+0000 7fc7266c8640 10 monclient: shutdown 2024-03-29T10:51:12.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.185+0000 7fc7266c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:12.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.185+0000 7fc7266c8640 1 -- 172.21.15.118:0/1284723971 >> v1:172.21.15.118:6789/0 conn(0x7fc7200b0540 legacy=0x7fc7201341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:12.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.185+0000 7fc7266c8640 1 -- 172.21.15.118:0/1284723971 shutdown_connections 2024-03-29T10:51:12.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.185+0000 7fc7266c8640 1 -- 172.21.15.118:0/1284723971 >> 172.21.15.118:0/1284723971 conn(0x7fc720066b00 msgr2=0x7fc72013a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:12.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.185+0000 7fc7266c8640 1 -- 172.21.15.118:0/1284723971 shutdown_connections 2024-03-29T10:51:12.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.185+0000 7fc7266c8640 1 -- 172.21.15.118:0/1284723971 wait complete. 2024-03-29T10:51:12.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-29T10:51:12.185+0000 7fc7266c8640 1 librados: shutdown 2024-03-29T10:51:12.177 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T10:51:12.178 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T10:51:12.182 INFO:tasks.ceph.osd.0:Started 2024-03-29T10:51:12.182 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T10:51:12.183 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T10:51:12.184 INFO:tasks.ceph.osd.1:Started 2024-03-29T10:51:12.185 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T10:51:12.185 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T10:51:12.185 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:12.203+0000 7f03b67b6040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:51:12.187 INFO:tasks.ceph.osd.2:Started 2024-03-29T10:51:12.187 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T10:51:12.187 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T10:51:12.189 INFO:tasks.ceph.osd.3:Started 2024-03-29T10:51:12.189 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T10:51:12.189 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T10:51:12.191 INFO:tasks.ceph.osd.4:Started 2024-03-29T10:51:12.191 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T10:51:12.191 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T10:51:12.193 INFO:tasks.ceph.osd.5:Started 2024-03-29T10:51:12.194 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T10:51:12.194 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T10:51:12.196 INFO:tasks.ceph.osd.6:Started 2024-03-29T10:51:12.196 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T10:51:12.196 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T10:51:12.198 INFO:tasks.ceph.osd.7:Started 2024-03-29T10:51:12.198 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:12.279 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:12.297+0000 7f0c627b8040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:51:12.328 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:12.345+0000 7f0c627b8040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:51:12.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.346+0000 7f1ce5431640 10 monclient: get_monmap_and_config 2024-03-29T10:51:12.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.346+0000 7f1ce5431640 10 monclient: build_initial_monmap 2024-03-29T10:51:12.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.346+0000 7f1ce5431640 10 monclient: monmap: 2024-03-29T10:51:12.329 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-03-29T10:51:12.329 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:12.329 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-03-29T10:51:12.348182+0000 2024-03-29T10:51:12.329 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-03-29T10:51:12.348182+0000 2024-03-29T10:51:12.329 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:12.329 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:12.329 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:12.329 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:51:12.329 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:51:12.329 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-03-29T10:51:12.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.346+0000 7f1ce5431640 1 Processor -- start 2024-03-29T10:51:12.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.346+0000 7f1ce5431640 1 -- start start 2024-03-29T10:51:12.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.346+0000 7f1ce5431640 10 monclient: init 2024-03-29T10:51:12.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.347+0000 7f1ce5431640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:12.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.347+0000 7f1ce5431640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:51:12.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.347+0000 7f1ce5431640 10 monclient(hunting): picked mon.noname-c con 0x7f1ce009ac40 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:12.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.347+0000 7f1ce5431640 10 monclient(hunting): picked mon.noname-b con 0x7f1ce009c870 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:12.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.347+0000 7f1ce5431640 10 monclient(hunting): picked mon.noname-a con 0x7f1ce00b4360 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:12.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.347+0000 7f1ce5431640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce009dec0 con 0x7f1ce00b4360 2024-03-29T10:51:12.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.347+0000 7f1ce5431640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce009da30 con 0x7f1ce009ac40 2024-03-29T10:51:12.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.347+0000 7f1ce5431640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce00b39c0 con 0x7f1ce009c870 2024-03-29T10:51:12.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.347+0000 7f1ce5431640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:12.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.347+0000 7f1ce5431640 10 monclient(hunting): authenticate will time out at 827.652652s 2024-03-29T10:51:12.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.347+0000 7f1cdf7fe640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f1ce00b4360 0x7f1ce014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:40750/0 (socket says 172.21.15.112:40750) 2024-03-29T10:51:12.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.347+0000 7f1cdf7fe640 1 -- 172.21.15.112:0/789781634 learned_addr learned my addr 172.21.15.112:0/789781634 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:51:12.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.347+0000 7f1cddffb640 1 -- 172.21.15.112:0/789781634 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3940997175 0 0) 0x7f1cd8001490 con 0x7f1ce00b4360 2024-03-29T10:51:12.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.347+0000 7f1cddffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:12.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.347+0000 7f1cddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:12.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.347+0000 7f1cddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:12.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.347+0000 7f1cddffb640 10 monclient(hunting): my global_id is 4179 2024-03-29T10:51:12.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.347+0000 7f1cddffb640 1 -- 172.21.15.112:0/789781634 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cc8003520 con 0x7f1ce00b4360 2024-03-29T10:51:12.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.347+0000 7f1cddffb640 1 -- 172.21.15.112:0/789781634 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813409236 0 0) 0x7f1ce009da30 con 0x7f1ce009ac40 2024-03-29T10:51:12.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:12.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:12.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:12.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 10 monclient(hunting): my global_id is 4186 2024-03-29T10:51:12.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 1 -- 172.21.15.112:0/789781634 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd8001490 con 0x7f1ce009ac40 2024-03-29T10:51:12.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 1 -- 172.21.15.112:0/789781634 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 614685744 0 0) 0x7f1cc8003520 con 0x7f1ce00b4360 2024-03-29T10:51:12.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 1 -- 172.21.15.112:0/789781634 >> v1:172.21.15.112:6790/0 conn(0x7f1ce009c870 legacy=0x7f1ce013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:12.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 1 -- 172.21.15.112:0/789781634 >> v1:172.21.15.118:6789/0 conn(0x7f1ce009ac40 legacy=0x7f1ce00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:12.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 10 monclient: _finish_hunting 0 2024-03-29T10:51:12.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 1 monclient: found mon.noname-a 2024-03-29T10:51:12.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:12.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T10:51:12.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 1 -- 172.21.15.112:0/789781634 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce00b5160 con 0x7f1ce00b4360 2024-03-29T10:51:12.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 10 monclient: _finish_auth 0 2024-03-29T10:51:12.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 10 monclient: _check_auth_tickets 2024-03-29T10:51:12.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:12.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 1 -- 172.21.15.112:0/789781634 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1cd8001a00 con 0x7f1ce00b4360 2024-03-29T10:51:12.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1ce5431640 5 monclient: authenticate success, global_id 4179 2024-03-29T10:51:12.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1ce5431640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:12.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:12.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:12.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 10 monclient: dump: 2024-03-29T10:51:12.343 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-03-29T10:51:12.343 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:12.344 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:12.344 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:12.344 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:12.344 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:12.344 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:12.344 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:12.344 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:12.344 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-03-29T10:51:12.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1ce5431640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:12.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 1 -- 172.21.15.112:0/789781634 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1cd8001be0 con 0x7f1ce00b4360 2024-03-29T10:51:12.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:12.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 1 -- 172.21.15.112:0/789781634 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1cd8001ff0 con 0x7f1ce00b4360 2024-03-29T10:51:12.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:12.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:12.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1cddffb640 10 monclient: dump: 2024-03-29T10:51:12.345 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-03-29T10:51:12.345 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:12.345 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:12.345 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:12.345 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:12.345 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:12.345 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:12.345 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:12.345 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:12.345 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-03-29T10:51:12.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1ce5431640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:12.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1ce5431640 4 set_mon_vals no callback set 2024-03-29T10:51:12.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1ce5431640 10 monclient: shutdown 2024-03-29T10:51:12.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1ce5431640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:12.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.348+0000 7f1ce5431640 1 -- 172.21.15.112:0/789781634 >> v1:172.21.15.112:6789/0 conn(0x7f1ce00b4360 legacy=0x7f1ce014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:12.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.349+0000 7f1ce5431640 1 -- 172.21.15.112:0/789781634 shutdown_connections 2024-03-29T10:51:12.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.349+0000 7f1ce5431640 1 -- 172.21.15.112:0/789781634 >> 172.21.15.112:0/789781634 conn(0x7f1ce0066b00 msgr2=0x7f1ce0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:12.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.349+0000 7f1ce5431640 1 -- 172.21.15.112:0/789781634 shutdown_connections 2024-03-29T10:51:12.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.349+0000 7f1ce5431640 1 -- 172.21.15.112:0/789781634 wait complete. 2024-03-29T10:51:12.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.349+0000 7f1ce5431640 10 monclient: build_initial_monmap 2024-03-29T10:51:12.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.349+0000 7f1ce5431640 10 monclient: monmap: 2024-03-29T10:51:12.346 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-03-29T10:51:12.346 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:12.346 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-03-29T10:51:12.346 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-03-29T10:51:12.346 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:12.347 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:12.347 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:12.347 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:51:12.347 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:51:12.347 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-03-29T10:51:12.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.349+0000 7f1ce5431640 1 librados: starting msgr at 2024-03-29T10:51:12.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.349+0000 7f1ce5431640 1 librados: starting objecter 2024-03-29T10:51:12.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.350+0000 7f1ce5431640 1 Processor -- start 2024-03-29T10:51:12.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.350+0000 7f1ce5431640 1 -- start start 2024-03-29T10:51:12.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.350+0000 7f1ce5431640 1 librados: setting wanted keys 2024-03-29T10:51:12.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.350+0000 7f1ce5431640 1 librados: calling monclient init 2024-03-29T10:51:12.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.350+0000 7f1ce5431640 10 monclient: init 2024-03-29T10:51:12.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.350+0000 7f1ce5431640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:12.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.350+0000 7f1ce5431640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:51:12.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.350+0000 7f1ce5431640 10 monclient(hunting): picked mon.noname-c con 0x7f1ce009ac40 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:12.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.350+0000 7f1ce5431640 10 monclient(hunting): picked mon.noname-b con 0x7f1ce01400a0 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:12.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.350+0000 7f1ce5431640 10 monclient(hunting): picked mon.noname-a con 0x7f1ce01d7540 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:12.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.350+0000 7f1ce5431640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce0140f30 con 0x7f1ce01d7540 2024-03-29T10:51:12.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.350+0000 7f1ce5431640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce00b0490 con 0x7f1ce01400a0 2024-03-29T10:51:12.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.350+0000 7f1ce5431640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce01420f0 con 0x7f1ce009ac40 2024-03-29T10:51:12.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.350+0000 7f1ce5431640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:12.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.350+0000 7f1ce5431640 10 monclient(hunting): authenticate will time out at 827.655908s 2024-03-29T10:51:12.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.350+0000 7f1cdf7fe640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f1ce01d7540 0x7f1ce01e17e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:40754/0 (socket says 172.21.15.112:40754) 2024-03-29T10:51:12.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.350+0000 7f1cdeffd640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f1ce009ac40 0x7f1ce0140820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59250/0 (socket says 172.21.15.112:59250) 2024-03-29T10:51:12.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.350+0000 7f1cdf7fe640 1 -- 172.21.15.112:0/3769851381 learned_addr learned my addr 172.21.15.112:0/3769851381 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:51:12.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.350+0000 7f1cc77fe640 10 client.?.objecter ms_handle_connect 0x7f1ce01d7540 2024-03-29T10:51:12.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.350+0000 7f1cc77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:12.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.350+0000 7f1cc77fe640 10 client.?.objecter ms_handle_connect 0x7f1ce009ac40 2024-03-29T10:51:12.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.350+0000 7f1cc77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:12.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.350+0000 7f1cc77fe640 10 client.?.objecter ms_handle_connect 0x7f1ce01400a0 2024-03-29T10:51:12.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.350+0000 7f1cc77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:12.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 892312039 0 0) 0x7f1ce00b39c0 con 0x7f1ce01400a0 2024-03-29T10:51:12.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:12.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:12.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:12.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient(hunting): my global_id is 4189 2024-03-29T10:51:12.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ccc004eb0 con 0x7f1ce01400a0 2024-03-29T10:51:12.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202087939 0 0) 0x7f1ce01420f0 con 0x7f1ce009ac40 2024-03-29T10:51:12.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:12.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:12.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:12.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient(hunting): my global_id is 4181 2024-03-29T10:51:12.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce00b39c0 con 0x7f1ce009ac40 2024-03-29T10:51:12.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510669066 0 0) 0x7f1ce0140f30 con 0x7f1ce01d7540 2024-03-29T10:51:12.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:12.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:12.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:12.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient(hunting): my global_id is 4182 2024-03-29T10:51:12.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce01420f0 con 0x7f1ce01d7540 2024-03-29T10:51:12.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3404315957 0 0) 0x7f1ccc004eb0 con 0x7f1ce01400a0 2024-03-29T10:51:12.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ce0140f30 con 0x7f1ce01400a0 2024-03-29T10:51:12.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1cd4001720 con 0x7f1ce01400a0 2024-03-29T10:51:12.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:12.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:12.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient(hunting): dump: 2024-03-29T10:51:12.351 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-03-29T10:51:12.351 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:12.351 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:12.351 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:12.351 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:12.351 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:12.351 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:12.351 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:12.351 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:12.351 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-03-29T10:51:12.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:12.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3782819493 0 0) 0x7f1ce01420f0 con 0x7f1ce01d7540 2024-03-29T10:51:12.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ccc004eb0 con 0x7f1ce01d7540 2024-03-29T10:51:12.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2645294740 0 0) 0x7f1ce00b39c0 con 0x7f1ce009ac40 2024-03-29T10:51:12.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ce01420f0 con 0x7f1ce009ac40 2024-03-29T10:51:12.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1cd00018b0 con 0x7f1ce009ac40 2024-03-29T10:51:12.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:12.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:12.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient(hunting): dump: 2024-03-29T10:51:12.352 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-03-29T10:51:12.352 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:12.352 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:12.352 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:12.352 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:12.352 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:12.352 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:12.353 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:12.353 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:12.353 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-03-29T10:51:12.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1cd8004b60 con 0x7f1ce01d7540 2024-03-29T10:51:12.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:12.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:12.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient(hunting): dump: 2024-03-29T10:51:12.353 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-03-29T10:51:12.353 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:12.353 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:12.353 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:12.353 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:12.353 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:12.353 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:12.353 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:12.354 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:12.354 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-03-29T10:51:12.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 270742229 0 0) 0x7f1ce0140f30 con 0x7f1ce01400a0 2024-03-29T10:51:12.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 >> v1:172.21.15.112:6790/0 conn(0x7f1ce009ac40 legacy=0x7f1ce0140820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:12.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 >> v1:172.21.15.112:6789/0 conn(0x7f1ce01d7540 legacy=0x7f1ce01e17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:12.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:51:12.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 1 monclient: found mon.b 2024-03-29T10:51:12.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:12.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:12.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce01d79f0 con 0x7f1ce01400a0 2024-03-29T10:51:12.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient: _finish_auth 0 2024-03-29T10:51:12.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:51:12.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:12.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1ce5431640 5 monclient: authenticate success, global_id 4189 2024-03-29T10:51:12.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1ce5431640 10 monclient: _renew_subs 2024-03-29T10:51:12.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1ce5431640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:12.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1ce5431640 1 -- 172.21.15.112:0/3769851381 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce01e2f00 con 0x7f1ce01400a0 2024-03-29T10:51:12.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1ce5431640 10 client.4189.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:12.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1cd4001cb0 con 0x7f1ce01400a0 2024-03-29T10:51:12.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:12.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1cd4003790 con 0x7f1ce01400a0 2024-03-29T10:51:12.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:12.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:12.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 10 monclient: dump: 2024-03-29T10:51:12.355 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-03-29T10:51:12.355 INFO:teuthology.orchestra.run.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:12.355 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:12.355 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:12.355 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:12.355 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:12.356 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:12.356 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:12.356 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:12.356 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-03-29T10:51:12.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cdd7fa640 4 set_mon_vals no callback set 2024-03-29T10:51:12.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1ce5431640 10 monclient: _renew_subs 2024-03-29T10:51:12.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1ce5431640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:12.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1ce5431640 1 -- 172.21.15.112:0/3769851381 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce01e3270 con 0x7f1ce01400a0 2024-03-29T10:51:12.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1cd4003cc0 con 0x7f1ce01400a0 2024-03-29T10:51:12.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.351+0000 7f1ce5431640 1 librados: init done 2024-03-29T10:51:12.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.352+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f1cd4004640 con 0x7f1ce01400a0 2024-03-29T10:51:12.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.352+0000 7f1cc77fe640 10 client.4189.objecter ms_dispatch 0x7f1ce0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:12.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.352+0000 7f1cc77fe640 3 client.4189.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:12.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.352+0000 7f1cc77fe640 3 client.4189.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:12.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.352+0000 7f1cc77fe640 20 client.4189.objecter dump_active .. 0 homeless 2024-03-29T10:51:12.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.352+0000 7f1ce5431640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:12.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.352+0000 7f1ce5431640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:12.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.352+0000 7f1ce5431640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:12.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.352+0000 7f1ce5431640 1 -- 172.21.15.112:0/3769851381 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce00b1ab0 con 0x7f1ce01400a0 2024-03-29T10:51:12.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.354+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1cd4004800 con 0x7f1ce01400a0 2024-03-29T10:51:12.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.354+0000 7f1cc77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:12.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.354+0000 7f1cc77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:12.357 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:12.355+0000 7f03b67b6040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:51:12.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.415+0000 7f1ce5431640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:12.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.415+0000 7f1ce5431640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:12.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.415+0000 7f1ce5431640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:12.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.415+0000 7f1ce5431640 1 -- 172.21.15.112:0/3769851381 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ce004f730 con 0x7f1ce01400a0 2024-03-29T10:51:12.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.415+0000 7f1cc77fe640 1 -- 172.21.15.112:0/3769851381 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f1cd4003f70 con 0x7f1ce01400a0 2024-03-29T10:51:12.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.415+0000 7f1cc77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:12.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.415+0000 7f1cc77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:12.399 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-03-29T10:51:12.399 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:12.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.417+0000 7f1ce5431640 10 librados: watch_flush enter 2024-03-29T10:51:12.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.417+0000 7f1ce5431640 10 librados: watch_flush exit 2024-03-29T10:51:12.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.417+0000 7f1ce5431640 20 client.4189.objecter shutdown clearing up homeless session... 2024-03-29T10:51:12.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.417+0000 7f1ce5431640 10 client.4189.objecter successfully canceled tick 2024-03-29T10:51:12.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.417+0000 7f1ce5431640 10 monclient: shutdown 2024-03-29T10:51:12.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.417+0000 7f1ce5431640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:12.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.417+0000 7f1ce5431640 1 -- 172.21.15.112:0/3769851381 >> v1:172.21.15.118:6789/0 conn(0x7f1ce01400a0 legacy=0x7f1ce01d6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:12.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.417+0000 7f1ce5431640 1 -- 172.21.15.112:0/3769851381 shutdown_connections 2024-03-29T10:51:12.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.417+0000 7f1ce5431640 1 -- 172.21.15.112:0/3769851381 >> 172.21.15.112:0/3769851381 conn(0x7f1ce0066b00 msgr2=0x7f1ce013a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:12.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.417+0000 7f1ce5431640 1 -- 172.21.15.112:0/3769851381 shutdown_connections 2024-03-29T10:51:12.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.417+0000 7f1ce5431640 1 -- 172.21.15.112:0/3769851381 wait complete. 2024-03-29T10:51:12.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-03-29T10:51:12.417+0000 7f1ce5431640 1 librados: shutdown 2024-03-29T10:51:12.410 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T10:51:12.410 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T10:51:12.446 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:12.463+0000 7f0c627b8040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:51:12.449 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:12.467+0000 7f03b67b6040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:51:12.493 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:12.512+0000 7f03b67b6040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:51:12.542 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:12.559+0000 7f0c627b8040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:51:12.607 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:12.625+0000 7f03b67b6040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:51:12.697 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:12.715+0000 7f0c627b8040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:51:12.701 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:12.719+0000 7f03b67b6040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:51:12.739 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:12.757+0000 7f0c627b8040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:51:12.796 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-03-29T10:51:12.814+0000 7fb9b3617640 -1 Falling back to public interface 2024-03-29T10:51:12.799 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-03-29T10:51:12.817+0000 7f51ee2d9640 -1 Falling back to public interface 2024-03-29T10:51:12.801 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-03-29T10:51:12.819+0000 7fbcbe6f5640 -1 Falling back to public interface 2024-03-29T10:51:12.805 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-03-29T10:51:12.822+0000 7f2917bac640 -1 Falling back to public interface 2024-03-29T10:51:12.810 INFO:tasks.ceph.osd.4.smithi118.stderr:2024-03-29T10:51:12.828+0000 7f02e2b73640 -1 Falling back to public interface 2024-03-29T10:51:12.813 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-03-29T10:51:12.830+0000 7f823e411640 -1 Falling back to public interface 2024-03-29T10:51:12.813 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-03-29T10:51:12.830+0000 7f139a21f640 -1 Falling back to public interface 2024-03-29T10:51:12.818 INFO:tasks.ceph.osd.7.smithi118.stderr:2024-03-29T10:51:12.836+0000 7fd605ea6640 -1 Falling back to public interface 2024-03-29T10:51:12.860 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:12.878+0000 7f03b67b6040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:51:12.898 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:12.915+0000 7f0c627b8040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:51:12.902 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:12.920+0000 7f03b67b6040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:51:13.054 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:13.071+0000 7f03b67b6040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:51:13.215 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:13.305 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.322+0000 7f09ef0ff640 10 monclient: get_monmap_and_config 2024-03-29T10:51:13.305 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.322+0000 7f09ef0ff640 10 monclient: build_initial_monmap 2024-03-29T10:51:13.305 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.322+0000 7f09ef0ff640 10 monclient: monmap: 2024-03-29T10:51:13.305 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:51:13.306 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:13.306 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:51:13.324804+0000 2024-03-29T10:51:13.306 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:51:13.324804+0000 2024-03-29T10:51:13.306 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:13.306 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:13.306 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:13.306 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:51:13.306 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:51:13.306 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:13.315 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.323+0000 7f09ef0ff640 1 Processor -- start 2024-03-29T10:51:13.315 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.323+0000 7f09ef0ff640 1 -- start start 2024-03-29T10:51:13.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.323+0000 7f09ef0ff640 10 monclient: init 2024-03-29T10:51:13.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09ef0ff640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:13.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09ef0ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:51:13.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09ef0ff640 10 monclient(hunting): picked mon.noname-a con 0x7f09e81457f0 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:13.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09ef0ff640 10 monclient(hunting): picked mon.noname-b con 0x7f09e813f650 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:13.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09ef0ff640 10 monclient(hunting): picked mon.noname-c con 0x7f09e80a9d70 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:13.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09ef0ff640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e80aaef0 con 0x7f09e81457f0 2024-03-29T10:51:13.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09ef0ff640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e8145bd0 con 0x7f09e80a9d70 2024-03-29T10:51:13.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09ef0ff640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e8145eb0 con 0x7f09e813f650 2024-03-29T10:51:13.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09ef0ff640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:13.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09ef0ff640 10 monclient(hunting): authenticate will time out at 828.629451s 2024-03-29T10:51:13.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09ece9b640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f09e81457f0 0x7f09e813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:40758/0 (socket says 172.21.15.112:40758) 2024-03-29T10:51:13.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e7fff640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f09e813f650 0x7f09e80b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59260/0 (socket says 172.21.15.112:59260) 2024-03-29T10:51:13.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09ece9b640 1 -- 172.21.15.112:0/1485833011 learned_addr learned my addr 172.21.15.112:0/1485833011 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:51:13.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 1 -- 172.21.15.112:0/1485833011 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339844784 0 0) 0x7f09e80aaef0 con 0x7f09e81457f0 2024-03-29T10:51:13.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:13.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:13.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:13.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 10 monclient(hunting): my global_id is 4185 2024-03-29T10:51:13.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 1 -- 172.21.15.112:0/1485833011 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09d8003520 con 0x7f09e81457f0 2024-03-29T10:51:13.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 1 -- 172.21.15.112:0/1485833011 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602317991 0 0) 0x7f09e8145eb0 con 0x7f09e813f650 2024-03-29T10:51:13.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:13.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:13.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:13.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 10 monclient(hunting): my global_id is 4184 2024-03-29T10:51:13.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 1 -- 172.21.15.112:0/1485833011 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e80aaef0 con 0x7f09e813f650 2024-03-29T10:51:13.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 1 -- 172.21.15.112:0/1485833011 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981223370 0 0) 0x7f09e8145bd0 con 0x7f09e80a9d70 2024-03-29T10:51:13.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:13.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:13.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:13.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 10 monclient(hunting): my global_id is 4192 2024-03-29T10:51:13.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 1 -- 172.21.15.112:0/1485833011 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e8145eb0 con 0x7f09e80a9d70 2024-03-29T10:51:13.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 1 -- 172.21.15.112:0/1485833011 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2700519490 0 0) 0x7f09d8003520 con 0x7f09e81457f0 2024-03-29T10:51:13.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 1 -- 172.21.15.112:0/1485833011 >> v1:172.21.15.112:6790/0 conn(0x7f09e813f650 legacy=0x7f09e80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:13.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 1 -- 172.21.15.112:0/1485833011 >> v1:172.21.15.118:6789/0 conn(0x7f09e80a9d70 legacy=0x7f09e814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:13.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:51:13.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 1 monclient: found mon.noname-a 2024-03-29T10:51:13.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:13.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T10:51:13.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 1 -- 172.21.15.112:0/1485833011 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e8146340 con 0x7f09e81457f0 2024-03-29T10:51:13.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 10 monclient: _finish_auth 0 2024-03-29T10:51:13.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:51:13.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:13.320 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09ef0ff640 5 monclient: authenticate success, global_id 4185 2024-03-29T10:51:13.320 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 1 -- 172.21.15.112:0/1485833011 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f09dc001690 con 0x7f09e81457f0 2024-03-29T10:51:13.320 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09ef0ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:13.320 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:13.320 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:13.320 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09e77fe640 10 monclient: dump: 2024-03-29T10:51:13.320 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:13.320 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:13.320 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:13.320 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:13.321 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:13.321 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:13.321 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:13.321 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:13.321 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:13.321 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:13.321 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.324+0000 7f09ef0ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:13.321 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.325+0000 7f09e77fe640 1 -- 172.21.15.112:0/1485833011 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09dc001870 con 0x7f09e81457f0 2024-03-29T10:51:13.321 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.325+0000 7f09e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:13.321 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.325+0000 7f09ef0ff640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:13.321 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.325+0000 7f09ef0ff640 4 set_mon_vals no callback set 2024-03-29T10:51:13.321 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.325+0000 7f09e77fe640 1 -- 172.21.15.112:0/1485833011 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f09dc001cc0 con 0x7f09e81457f0 2024-03-29T10:51:13.322 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.325+0000 7f09e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:13.322 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.325+0000 7f09e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:13.322 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.325+0000 7f09e77fe640 10 monclient: dump: 2024-03-29T10:51:13.322 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:13.322 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:13.322 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:13.322 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:13.322 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:13.322 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:13.322 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:13.322 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:13.322 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:13.322 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:13.322 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.325+0000 7f09ef0ff640 10 monclient: shutdown 2024-03-29T10:51:13.323 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.325+0000 7f09ef0ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:13.323 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.325+0000 7f09ef0ff640 1 -- 172.21.15.112:0/1485833011 >> v1:172.21.15.112:6789/0 conn(0x7f09e81457f0 legacy=0x7f09e813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:13.323 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.325+0000 7f09ef0ff640 1 -- 172.21.15.112:0/1485833011 shutdown_connections 2024-03-29T10:51:13.323 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.325+0000 7f09ef0ff640 1 -- 172.21.15.112:0/1485833011 >> 172.21.15.112:0/1485833011 conn(0x7f09e8066a00 msgr2=0x7f09e8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:13.323 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.325+0000 7f09ef0ff640 1 -- 172.21.15.112:0/1485833011 shutdown_connections 2024-03-29T10:51:13.323 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.325+0000 7f09ef0ff640 1 -- 172.21.15.112:0/1485833011 wait complete. 2024-03-29T10:51:13.323 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.326+0000 7f09ef0ff640 10 monclient: build_initial_monmap 2024-03-29T10:51:13.323 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.326+0000 7f09ef0ff640 10 monclient: monmap: 2024-03-29T10:51:13.323 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:51:13.323 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:13.323 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:51:13.323 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:51:13.323 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:13.323 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:13.323 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:13.324 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:51:13.324 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:51:13.324 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:13.324 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.326+0000 7f09ef0ff640 1 librados: starting msgr at 2024-03-29T10:51:13.324 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.326+0000 7f09ef0ff640 1 librados: starting objecter 2024-03-29T10:51:13.324 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.326+0000 7f09ef0ff640 1 Processor -- start 2024-03-29T10:51:13.324 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.326+0000 7f09ef0ff640 1 -- start start 2024-03-29T10:51:13.324 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.326+0000 7f09ef0ff640 1 librados: setting wanted keys 2024-03-29T10:51:13.324 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.326+0000 7f09ef0ff640 1 librados: calling monclient init 2024-03-29T10:51:13.324 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.326+0000 7f09ef0ff640 10 monclient: init 2024-03-29T10:51:13.324 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09ef0ff640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:13.324 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09ef0ff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:13.324 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09ef0ff640 10 monclient(hunting): picked mon.noname-a con 0x7f09e80a9d70 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:13.324 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09ef0ff640 10 monclient(hunting): picked mon.noname-b con 0x7f09e813f650 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:13.325 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09ef0ff640 10 monclient(hunting): picked mon.noname-c con 0x7f09e81457f0 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:13.325 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09ef0ff640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e81db2d0 con 0x7f09e80a9d70 2024-03-29T10:51:13.325 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09ef0ff640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e813afd0 con 0x7f09e813f650 2024-03-29T10:51:13.325 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09ef0ff640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e80b1770 con 0x7f09e81457f0 2024-03-29T10:51:13.325 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09ef0ff640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:13.325 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09ef0ff640 10 monclient(hunting): authenticate will time out at 828.632644s 2024-03-29T10:51:13.325 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09ece9b640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f09e80a9d70 0x7f09e813b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:40760/0 (socket says 172.21.15.112:40760) 2024-03-29T10:51:13.325 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09ed69c640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f09e81457f0 0x7f09e80b0650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59268/0 (socket says 172.21.15.112:59268) 2024-03-29T10:51:13.325 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09ece9b640 1 -- 172.21.15.112:0/1992807988 learned_addr learned my addr 172.21.15.112:0/1992807988 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:51:13.325 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09e57fa640 10 client.?.objecter ms_handle_connect 0x7f09e81457f0 2024-03-29T10:51:13.325 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09e57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:13.325 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09e57fa640 10 client.?.objecter ms_handle_connect 0x7f09e80a9d70 2024-03-29T10:51:13.325 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09e57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:13.325 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09e57fa640 10 client.?.objecter ms_handle_connect 0x7f09e813f650 2024-03-29T10:51:13.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09e57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:13.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2796679546 0 0) 0x7f09e813afd0 con 0x7f09e813f650 2024-03-29T10:51:13.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09e57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:51:13.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:13.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:13.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09e57fa640 10 monclient(hunting): my global_id is 4195 2024-03-29T10:51:13.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09cc004eb0 con 0x7f09e813f650 2024-03-29T10:51:13.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2967463238 0 0) 0x7f09e80b1770 con 0x7f09e81457f0 2024-03-29T10:51:13.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09e57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:51:13.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:13.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:13.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09e57fa640 10 monclient(hunting): my global_id is 4187 2024-03-29T10:51:13.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e813afd0 con 0x7f09e81457f0 2024-03-29T10:51:13.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672439716 0 0) 0x7f09e81db2d0 con 0x7f09e80a9d70 2024-03-29T10:51:13.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09e57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:51:13.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:13.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:13.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09e57fa640 10 monclient(hunting): my global_id is 4188 2024-03-29T10:51:13.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.327+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e80b1770 con 0x7f09e80a9d70 2024-03-29T10:51:13.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2758282924 0 0) 0x7f09cc004eb0 con 0x7f09e813f650 2024-03-29T10:51:13.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09e81db2d0 con 0x7f09e813f650 2024-03-29T10:51:13.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f09d4001290 con 0x7f09e813f650 2024-03-29T10:51:13.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:13.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:13.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 10 monclient(hunting): dump: 2024-03-29T10:51:13.327 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:13.327 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:13.327 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:13.327 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:13.327 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:13.328 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:13.328 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:13.328 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:13.328 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:13.328 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:13.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:13.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2117880444 0 0) 0x7f09e813afd0 con 0x7f09e81457f0 2024-03-29T10:51:13.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09cc004eb0 con 0x7f09e81457f0 2024-03-29T10:51:13.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3020233515 0 0) 0x7f09e80b1770 con 0x7f09e80a9d70 2024-03-29T10:51:13.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09e813afd0 con 0x7f09e80a9d70 2024-03-29T10:51:13.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f09e0001bf0 con 0x7f09e81457f0 2024-03-29T10:51:13.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:13.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:13.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 10 monclient(hunting): dump: 2024-03-29T10:51:13.329 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:13.329 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:13.329 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:13.329 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:13.329 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:13.329 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:13.329 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:13.329 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:13.329 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:13.329 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:13.329 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f09dc001b80 con 0x7f09e80a9d70 2024-03-29T10:51:13.329 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:13.329 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:13.329 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 10 monclient(hunting): dump: 2024-03-29T10:51:13.329 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:13.330 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:13.330 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:13.330 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:13.330 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:13.330 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:13.330 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:13.330 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:13.330 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:13.330 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:13.330 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2667997042 0 0) 0x7f09e81db2d0 con 0x7f09e813f650 2024-03-29T10:51:13.330 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 >> v1:172.21.15.112:6790/0 conn(0x7f09e81457f0 legacy=0x7f09e80b0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:13.330 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 >> v1:172.21.15.112:6789/0 conn(0x7f09e80a9d70 legacy=0x7f09e813b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:13.330 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 10 monclient: _finish_hunting 0 2024-03-29T10:51:13.330 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 1 monclient: found mon.b 2024-03-29T10:51:13.330 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:13.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:13.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e81db760 con 0x7f09e813f650 2024-03-29T10:51:13.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 10 monclient: _finish_auth 0 2024-03-29T10:51:13.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 10 monclient: _check_auth_tickets 2024-03-29T10:51:13.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:13.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09ef0ff640 5 monclient: authenticate success, global_id 4195 2024-03-29T10:51:13.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09ef0ff640 10 monclient: _renew_subs 2024-03-29T10:51:13.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09ef0ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:13.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09ef0ff640 1 -- 172.21.15.112:0/1992807988 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e81391a0 con 0x7f09e813f650 2024-03-29T10:51:13.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09ef0ff640 10 client.4195.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:13.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09ef0ff640 10 monclient: _renew_subs 2024-03-29T10:51:13.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09ef0ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:13.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09ef0ff640 1 -- 172.21.15.112:0/1992807988 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e80b0f40 con 0x7f09e813f650 2024-03-29T10:51:13.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09ef0ff640 1 librados: init done 2024-03-29T10:51:13.332 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09d40047e0 con 0x7f09e813f650 2024-03-29T10:51:13.332 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:13.332 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f09d4004af0 con 0x7f09e813f650 2024-03-29T10:51:13.332 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:13.332 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:13.332 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e6ffd640 4 set_mon_vals no callback set 2024-03-29T10:51:13.332 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 10 monclient: dump: 2024-03-29T10:51:13.332 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:13.332 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:13.332 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:13.332 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:13.332 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:13.332 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:13.332 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:13.332 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:13.333 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:13.333 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:13.333 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f09d4004fe0 con 0x7f09e813f650 2024-03-29T10:51:13.333 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f09d4006940 con 0x7f09e813f650 2024-03-29T10:51:13.333 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 10 client.4195.objecter ms_dispatch 0x7f09e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:13.333 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 3 client.4195.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:13.333 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 3 client.4195.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:13.333 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.328+0000 7f09e57fa640 20 client.4195.objecter dump_active .. 0 homeless 2024-03-29T10:51:13.333 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.329+0000 7f09ef0ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:13.333 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.329+0000 7f09ef0ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:13.333 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.329+0000 7f09ef0ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:13.333 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.329+0000 7f09ef0ff640 1 -- 172.21.15.112:0/1992807988 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09c8005080 con 0x7f09e813f650 2024-03-29T10:51:13.333 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.331+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f09d4006bd0 con 0x7f09e813f650 2024-03-29T10:51:13.333 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.331+0000 7f09e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:13.334 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.331+0000 7f09e57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:13.375 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.392+0000 7f09ef0ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:13.375 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.392+0000 7f09ef0ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:13.375 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.392+0000 7f09ef0ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:13.375 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.392+0000 7f09ef0ff640 1 -- 172.21.15.112:0/1992807988 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09c8005330 con 0x7f09e813f650 2024-03-29T10:51:13.375 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.393+0000 7f09e57fa640 1 -- 172.21.15.112:0/1992807988 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f09d40063d0 con 0x7f09e813f650 2024-03-29T10:51:13.375 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.393+0000 7f09e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:13.376 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.393+0000 7f09e57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:13.376 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:51:13.376 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:13.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.393+0000 7f09ef0ff640 10 librados: watch_flush enter 2024-03-29T10:51:13.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.393+0000 7f09ef0ff640 10 librados: watch_flush exit 2024-03-29T10:51:13.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.393+0000 7f09ef0ff640 20 client.4195.objecter shutdown clearing up homeless session... 2024-03-29T10:51:13.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.393+0000 7f09ef0ff640 10 client.4195.objecter successfully canceled tick 2024-03-29T10:51:13.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.393+0000 7f09ef0ff640 10 monclient: shutdown 2024-03-29T10:51:13.378 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.393+0000 7f09ef0ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:13.378 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.393+0000 7f09ef0ff640 1 -- 172.21.15.112:0/1992807988 >> v1:172.21.15.118:6789/0 conn(0x7f09e813f650 legacy=0x7f09e8138a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:13.378 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.393+0000 7f09ef0ff640 1 -- 172.21.15.112:0/1992807988 shutdown_connections 2024-03-29T10:51:13.378 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.393+0000 7f09ef0ff640 1 -- 172.21.15.112:0/1992807988 >> 172.21.15.112:0/1992807988 conn(0x7f09e8066a00 msgr2=0x7f09e8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:13.378 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.393+0000 7f09ef0ff640 1 -- 172.21.15.112:0/1992807988 shutdown_connections 2024-03-29T10:51:13.378 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.393+0000 7f09ef0ff640 1 -- 172.21.15.112:0/1992807988 wait complete. 2024-03-29T10:51:13.378 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:13.394+0000 7f09ef0ff640 1 librados: shutdown 2024-03-29T10:51:13.385 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:13.881 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:13.899+0000 7f03b67b6040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:51:14.017 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:14.035+0000 7f0c627b8040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:51:14.156 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:14.174+0000 7f03b67b6040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:51:14.210 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:14.228+0000 7f03b67b6040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:51:14.290 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:14.308+0000 7f0c627b8040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:51:14.324 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:14.343+0000 7f03b67b6040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:51:14.342 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:14.360+0000 7f0c627b8040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:51:14.461 INFO:tasks.ceph.mgr.y.smithi112.stderr:2024-03-29T10:51:14.479+0000 7f0c627b8040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:51:15.337 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:15.355+0000 7f0357c17640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T10:51:15.338 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-03-29T10:51:15.356+0000 7f0357c17640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T10:51:16.268 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-03-29T10:51:16.286+0000 7f2917bac640 -1 osd.0 0 log_to_monitors true 2024-03-29T10:51:16.279 INFO:tasks.ceph.osd.6.smithi118.stderr:2024-03-29T10:51:16.297+0000 7fb9b3617640 -1 osd.6 0 log_to_monitors true 2024-03-29T10:51:16.313 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-03-29T10:51:16.331+0000 7fbcbe6f5640 -1 osd.1 0 log_to_monitors true 2024-03-29T10:51:16.324 INFO:tasks.ceph.osd.7.smithi118.stderr:2024-03-29T10:51:16.342+0000 7fd605ea6640 -1 osd.7 0 log_to_monitors true 2024-03-29T10:51:16.325 INFO:tasks.ceph.osd.3.smithi112.stderr:2024-03-29T10:51:16.342+0000 7f823e411640 -1 osd.3 0 log_to_monitors true 2024-03-29T10:51:16.342 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-03-29T10:51:16.360+0000 7f51ee2d9640 -1 osd.5 0 log_to_monitors true 2024-03-29T10:51:16.345 INFO:tasks.ceph.osd.4.smithi118.stderr:2024-03-29T10:51:16.363+0000 7f02e2b73640 -1 osd.4 0 log_to_monitors true 2024-03-29T10:51:16.349 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-03-29T10:51:16.366+0000 7f139a21f640 -1 osd.2 0 log_to_monitors true 2024-03-29T10:51:20.191 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:20.356 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.375+0000 7fd294b3b640 10 monclient: get_monmap_and_config 2024-03-29T10:51:20.356 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.375+0000 7fd294b3b640 10 monclient: build_initial_monmap 2024-03-29T10:51:20.356 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.375+0000 7fd294b3b640 10 monclient: monmap: 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:51:20.376048+0000 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:51:20.376048+0000 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:20.358 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.376+0000 7fd294b3b640 1 Processor -- start 2024-03-29T10:51:20.358 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.376+0000 7fd294b3b640 1 -- start start 2024-03-29T10:51:20.358 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.376+0000 7fd294b3b640 10 monclient: init 2024-03-29T10:51:20.364 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.377+0000 7fd294b3b640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:20.364 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.377+0000 7fd294b3b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:20.364 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.377+0000 7fd294b3b640 10 monclient(hunting): picked mon.noname-a con 0x7fd2901422b0 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:20.364 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.377+0000 7fd294b3b640 10 monclient(hunting): picked mon.noname-c con 0x7fd290142690 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:20.364 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.377+0000 7fd28dd9b640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fd290142690 0x7fd2900b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:46576/0 (socket says 172.21.15.112:46576) 2024-03-29T10:51:20.364 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.377+0000 7fd28ed9d640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fd2900b17d0 0x7fd2900b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:46244/0 (socket says 172.21.15.112:46244) 2024-03-29T10:51:20.364 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.377+0000 7fd28ed9d640 1 -- 172.21.15.112:0/2721009827 learned_addr learned my addr 172.21.15.112:0/2721009827 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:51:20.365 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.377+0000 7fd294b3b640 10 monclient(hunting): picked mon.noname-b con 0x7fd2900b17d0 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:20.365 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.377+0000 7fd294b3b640 1 -- 172.21.15.112:0/2721009827 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd290142c80 con 0x7fd2901422b0 2024-03-29T10:51:20.365 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.378+0000 7fd294b3b640 1 -- 172.21.15.112:0/2721009827 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2900aae40 con 0x7fd290142690 2024-03-29T10:51:20.365 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.378+0000 7fd294b3b640 1 -- 172.21.15.112:0/2721009827 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29013fbd0 con 0x7fd2900b17d0 2024-03-29T10:51:20.365 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.378+0000 7fd294b3b640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:20.365 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.378+0000 7fd294b3b640 10 monclient(hunting): authenticate will time out at 835.682536s 2024-03-29T10:51:20.365 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.378+0000 7fd28d59a640 1 -- 172.21.15.112:0/2721009827 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673199011 0 0) 0x7fd29013fbd0 con 0x7fd2900b17d0 2024-03-29T10:51:20.365 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.378+0000 7fd28d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:51:20.365 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.378+0000 7fd28d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:20.365 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.378+0000 7fd28d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:20.365 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.378+0000 7fd28d59a640 10 monclient(hunting): my global_id is 4229 2024-03-29T10:51:20.365 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.378+0000 7fd28d59a640 1 -- 172.21.15.112:0/2721009827 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd278003520 con 0x7fd2900b17d0 2024-03-29T10:51:20.365 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.378+0000 7fd28d59a640 1 -- 172.21.15.112:0/2721009827 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009032080 0 0) 0x7fd2900aae40 con 0x7fd290142690 2024-03-29T10:51:20.365 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.378+0000 7fd28d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:51:20.366 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.378+0000 7fd28d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:20.366 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.378+0000 7fd28d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:20.366 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.378+0000 7fd28d59a640 10 monclient(hunting): my global_id is 4240 2024-03-29T10:51:20.366 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.378+0000 7fd28d59a640 1 -- 172.21.15.112:0/2721009827 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd29013fbd0 con 0x7fd290142690 2024-03-29T10:51:20.366 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.379+0000 7fd28d59a640 1 -- 172.21.15.112:0/2721009827 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2581626316 0 0) 0x7fd278003520 con 0x7fd2900b17d0 2024-03-29T10:51:20.366 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.379+0000 7fd28d59a640 1 -- 172.21.15.112:0/2721009827 >> v1:172.21.15.118:6789/0 conn(0x7fd290142690 legacy=0x7fd2900b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:20.366 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.379+0000 7fd28d59a640 1 -- 172.21.15.112:0/2721009827 >> v1:172.21.15.112:6789/0 conn(0x7fd2901422b0 legacy=0x7fd29013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:20.366 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.379+0000 7fd28d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:51:20.366 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.379+0000 7fd28d59a640 1 monclient: found mon.noname-b 2024-03-29T10:51:20.366 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.379+0000 7fd28d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:20.366 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.379+0000 7fd28d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:51:20.366 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.379+0000 7fd28d59a640 1 -- 172.21.15.112:0/2721009827 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29014f920 con 0x7fd2900b17d0 2024-03-29T10:51:20.366 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.379+0000 7fd28d59a640 10 monclient: _finish_auth 0 2024-03-29T10:51:20.367 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.379+0000 7fd28d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:51:20.367 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.379+0000 7fd28d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:20.367 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.379+0000 7fd28d59a640 1 -- 172.21.15.112:0/2721009827 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fd288001a10 con 0x7fd2900b17d0 2024-03-29T10:51:20.367 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.379+0000 7fd28d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:20.367 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.379+0000 7fd28d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:20.367 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.379+0000 7fd28d59a640 10 monclient: dump: 2024-03-29T10:51:20.367 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:20.367 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:20.367 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:20.367 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:20.367 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:20.367 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:20.367 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:20.367 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:20.368 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:20.368 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:20.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.379+0000 7fd294b3b640 5 monclient: authenticate success, global_id 4229 2024-03-29T10:51:20.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.379+0000 7fd294b3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:20.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.379+0000 7fd28d59a640 1 -- 172.21.15.112:0/2721009827 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd288001b70 con 0x7fd2900b17d0 2024-03-29T10:51:20.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.380+0000 7fd28d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:20.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.380+0000 7fd28d59a640 1 -- 172.21.15.112:0/2721009827 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fd288001f00 con 0x7fd2900b17d0 2024-03-29T10:51:20.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.380+0000 7fd294b3b640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:20.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.380+0000 7fd294b3b640 4 set_mon_vals no callback set 2024-03-29T10:51:20.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.380+0000 7fd28d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:20.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.380+0000 7fd28d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:20.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.380+0000 7fd28d59a640 10 monclient: dump: 2024-03-29T10:51:20.368 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:20.368 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.381+0000 7fd294b3b640 10 monclient: shutdown 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.381+0000 7fd294b3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.381+0000 7fd294b3b640 1 -- 172.21.15.112:0/2721009827 >> v1:172.21.15.112:6790/0 conn(0x7fd2900b17d0 legacy=0x7fd2900b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.381+0000 7fd294b3b640 1 -- 172.21.15.112:0/2721009827 shutdown_connections 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.381+0000 7fd294b3b640 1 -- 172.21.15.112:0/2721009827 >> 172.21.15.112:0/2721009827 conn(0x7fd290066a00 msgr2=0x7fd290066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.381+0000 7fd294b3b640 1 -- 172.21.15.112:0/2721009827 shutdown_connections 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.381+0000 7fd294b3b640 1 -- 172.21.15.112:0/2721009827 wait complete. 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.382+0000 7fd294b3b640 10 monclient: build_initial_monmap 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.383+0000 7fd294b3b640 10 monclient: monmap: 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:51:20.371 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:20.381 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.383+0000 7fd294b3b640 1 librados: starting msgr at 2024-03-29T10:51:20.381 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.383+0000 7fd294b3b640 1 librados: starting objecter 2024-03-29T10:51:20.381 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.383+0000 7fd294b3b640 1 Processor -- start 2024-03-29T10:51:20.381 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.384+0000 7fd294b3b640 1 -- start start 2024-03-29T10:51:20.381 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.384+0000 7fd294b3b640 1 librados: setting wanted keys 2024-03-29T10:51:20.381 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.384+0000 7fd294b3b640 1 librados: calling monclient init 2024-03-29T10:51:20.381 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.384+0000 7fd294b3b640 10 monclient: init 2024-03-29T10:51:20.382 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.385+0000 7fd294b3b640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:20.382 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.385+0000 7fd294b3b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:20.382 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.385+0000 7fd294b3b640 10 monclient(hunting): picked mon.noname-c con 0x7fd2900b17d0 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:20.382 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.385+0000 7fd294b3b640 10 monclient(hunting): picked mon.noname-a con 0x7fd2901422b0 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:20.382 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.385+0000 7fd28dd9b640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fd2901422b0 0x7fd2900714c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60430/0 (socket says 172.21.15.112:60430) 2024-03-29T10:51:20.382 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.385+0000 7fd28dd9b640 1 -- 172.21.15.112:0/1105902491 learned_addr learned my addr 172.21.15.112:0/1105902491 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:51:20.382 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.385+0000 7fd294b3b640 10 monclient(hunting): picked mon.noname-b con 0x7fd290142690 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:20.382 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.385+0000 7fd294b3b640 1 -- 172.21.15.112:0/1105902491 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29006ea70 con 0x7fd2901422b0 2024-03-29T10:51:20.382 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.385+0000 7fd294b3b640 1 -- 172.21.15.112:0/1105902491 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29006ed50 con 0x7fd290142690 2024-03-29T10:51:20.382 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.385+0000 7fd294b3b640 1 -- 172.21.15.112:0/1105902491 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2901df300 con 0x7fd2900b17d0 2024-03-29T10:51:20.382 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.385+0000 7fd294b3b640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:20.382 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.385+0000 7fd294b3b640 10 monclient(hunting): authenticate will time out at 835.689705s 2024-03-29T10:51:20.382 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.385+0000 7fd27effd640 10 client.?.objecter ms_handle_connect 0x7fd2901422b0 2024-03-29T10:51:20.383 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.385+0000 7fd27effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:20.383 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.385+0000 7fd27effd640 1 -- 172.21.15.112:0/1105902491 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813639919 0 0) 0x7fd29006ea70 con 0x7fd2901422b0 2024-03-29T10:51:20.383 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.385+0000 7fd27effd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:20.383 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.385+0000 7fd27effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:20.383 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.385+0000 7fd27effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:20.383 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.385+0000 7fd27effd640 10 monclient(hunting): my global_id is 4233 2024-03-29T10:51:20.383 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.385+0000 7fd27effd640 1 -- 172.21.15.112:0/1105902491 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd270004eb0 con 0x7fd2901422b0 2024-03-29T10:51:20.383 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.385+0000 7fd27effd640 10 client.?.objecter ms_handle_connect 0x7fd290142690 2024-03-29T10:51:20.383 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.385+0000 7fd27effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:20.383 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.386+0000 7fd27effd640 10 client.?.objecter ms_handle_connect 0x7fd2900b17d0 2024-03-29T10:51:20.383 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.386+0000 7fd27effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:20.383 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.386+0000 7fd27effd640 1 -- 172.21.15.112:0/1105902491 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726094956 0 0) 0x7fd29006ed50 con 0x7fd290142690 2024-03-29T10:51:20.383 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.386+0000 7fd27effd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:20.383 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.386+0000 7fd27effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:20.384 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.386+0000 7fd27effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:20.384 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.386+0000 7fd27effd640 10 monclient(hunting): my global_id is 4243 2024-03-29T10:51:20.384 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.386+0000 7fd27effd640 1 -- 172.21.15.112:0/1105902491 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd29006ea70 con 0x7fd290142690 2024-03-29T10:51:20.384 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.386+0000 7fd27effd640 1 -- 172.21.15.112:0/1105902491 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255705987 0 0) 0x7fd2901df300 con 0x7fd2900b17d0 2024-03-29T10:51:20.384 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.386+0000 7fd27effd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:20.384 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.386+0000 7fd27effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:20.384 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.386+0000 7fd27effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:20.384 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.386+0000 7fd27effd640 10 monclient(hunting): my global_id is 4232 2024-03-29T10:51:20.385 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.386+0000 7fd27effd640 1 -- 172.21.15.112:0/1105902491 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd29006ed50 con 0x7fd2900b17d0 2024-03-29T10:51:20.385 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.386+0000 7fd27effd640 1 -- 172.21.15.112:0/1105902491 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3442558816 0 0) 0x7fd29006ea70 con 0x7fd290142690 2024-03-29T10:51:20.385 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.386+0000 7fd27effd640 1 -- 172.21.15.112:0/1105902491 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2901df300 con 0x7fd290142690 2024-03-29T10:51:20.385 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.386+0000 7fd27effd640 1 -- 172.21.15.112:0/1105902491 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fd288001e80 con 0x7fd290142690 2024-03-29T10:51:20.385 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.386+0000 7fd27effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:20.385 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.386+0000 7fd27effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:20.385 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.386+0000 7fd27effd640 10 monclient(hunting): dump: 2024-03-29T10:51:20.385 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:20.385 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:20.385 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:20.385 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:20.385 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:20.385 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:20.385 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:20.386 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:20.386 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:20.386 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:20.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.386+0000 7fd27effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:20.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.387+0000 7fd27effd640 1 -- 172.21.15.112:0/1105902491 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3465039237 0 0) 0x7fd270004eb0 con 0x7fd2901422b0 2024-03-29T10:51:20.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.387+0000 7fd27effd640 1 -- 172.21.15.112:0/1105902491 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd29006ea70 con 0x7fd2901422b0 2024-03-29T10:51:20.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.387+0000 7fd27effd640 1 -- 172.21.15.112:0/1105902491 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fd280001930 con 0x7fd2901422b0 2024-03-29T10:51:20.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.387+0000 7fd27effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:20.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.387+0000 7fd27effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:20.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.387+0000 7fd27effd640 10 monclient(hunting): dump: 2024-03-29T10:51:20.386 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:20.386 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:20.386 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:20.387 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:20.387 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:20.387 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:20.387 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:20.387 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:20.387 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:20.387 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:20.387 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.387+0000 7fd27effd640 1 -- 172.21.15.112:0/1105902491 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 645956913 0 0) 0x7fd2901df300 con 0x7fd290142690 2024-03-29T10:51:20.387 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.387+0000 7fd27effd640 1 -- 172.21.15.112:0/1105902491 >> v1:172.21.15.112:6790/0 conn(0x7fd2900b17d0 legacy=0x7fd29012f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:20.387 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.387+0000 7fd27effd640 1 -- 172.21.15.112:0/1105902491 >> v1:172.21.15.112:6789/0 conn(0x7fd2901422b0 legacy=0x7fd2900714c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:20.387 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.387+0000 7fd27effd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:20.387 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.387+0000 7fd27effd640 1 monclient: found mon.b 2024-03-29T10:51:20.387 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.387+0000 7fd27effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:20.387 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.387+0000 7fd27effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:20.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.387+0000 7fd27effd640 1 -- 172.21.15.112:0/1105902491 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd290130010 con 0x7fd290142690 2024-03-29T10:51:20.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.387+0000 7fd27effd640 10 monclient: _finish_auth 0 2024-03-29T10:51:20.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.387+0000 7fd27effd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:20.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.387+0000 7fd27effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:20.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.388+0000 7fd294b3b640 5 monclient: authenticate success, global_id 4243 2024-03-29T10:51:20.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.388+0000 7fd294b3b640 10 monclient: _renew_subs 2024-03-29T10:51:20.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.388+0000 7fd294b3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:20.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.388+0000 7fd294b3b640 1 -- 172.21.15.112:0/1105902491 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2901df800 con 0x7fd290142690 2024-03-29T10:51:20.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.388+0000 7fd294b3b640 10 client.4243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:20.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.388+0000 7fd294b3b640 10 monclient: _renew_subs 2024-03-29T10:51:20.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.388+0000 7fd294b3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:20.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.388+0000 7fd294b3b640 1 -- 172.21.15.112:0/1105902491 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2900b6af0 con 0x7fd290142690 2024-03-29T10:51:20.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.388+0000 7fd294b3b640 1 librados: init done 2024-03-29T10:51:20.389 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.388+0000 7fd27effd640 1 -- 172.21.15.112:0/1105902491 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd288001460 con 0x7fd290142690 2024-03-29T10:51:20.389 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.388+0000 7fd27effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:20.389 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.388+0000 7fd27effd640 1 -- 172.21.15.112:0/1105902491 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fd288004fb0 con 0x7fd290142690 2024-03-29T10:51:20.389 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.388+0000 7fd27effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:20.389 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.388+0000 7fd27effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:20.389 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.388+0000 7fd27effd640 10 monclient: dump: 2024-03-29T10:51:20.389 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:20.389 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:20.389 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:20.389 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:20.389 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:20.389 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:20.389 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:20.389 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:20.390 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:20.390 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:20.390 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.388+0000 7fd28cd99640 4 set_mon_vals no callback set 2024-03-29T10:51:20.390 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.392+0000 7fd27effd640 1 -- 172.21.15.112:0/1105902491 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fd28801af80 con 0x7fd290142690 2024-03-29T10:51:20.390 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.393+0000 7fd27effd640 1 -- 172.21.15.112:0/1105902491 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7fd288084d70 con 0x7fd290142690 2024-03-29T10:51:20.390 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.393+0000 7fd27effd640 10 client.4243.objecter ms_dispatch 0x7fd290000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:20.390 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.393+0000 7fd27effd640 3 client.4243.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:20.390 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.393+0000 7fd27effd640 3 client.4243.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:20.390 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.393+0000 7fd27effd640 20 client.4243.objecter dump_active .. 0 homeless 2024-03-29T10:51:20.390 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.393+0000 7fd28e59c640 10 monclient: get_auth_request con 0x7fd27006de30 auth_method 0 2024-03-29T10:51:20.390 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.393+0000 7fd294b3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:20.390 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.393+0000 7fd294b3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:20.390 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.393+0000 7fd294b3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:20.391 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.393+0000 7fd294b3b640 1 -- 172.21.15.112:0/1105902491 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd274005080 con 0x7fd290142690 2024-03-29T10:51:20.391 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.394+0000 7fd27effd640 10 client.4243.objecter ms_handle_connect 0x7fd27006de30 2024-03-29T10:51:20.391 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.400+0000 7fd27effd640 1 -- 172.21.15.112:0/1105902491 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd288050650 con 0x7fd290142690 2024-03-29T10:51:20.391 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.400+0000 7fd27effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:20.391 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.400+0000 7fd27effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:20.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.762+0000 7fd294b3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:20.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.762+0000 7fd294b3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:20.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.762+0000 7fd294b3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:20.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.762+0000 7fd294b3b640 1 -- 172.21.15.112:0/1105902491 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd274005330 con 0x7fd290142690 2024-03-29T10:51:20.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.763+0000 7fd27effd640 1 -- 172.21.15.112:0/1105902491 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7fd288052580 con 0x7fd290142690 2024-03-29T10:51:20.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.763+0000 7fd27effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:20.745 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.763+0000 7fd27effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:20.745 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:51:20.745 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:20.749 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.767+0000 7fd294b3b640 10 librados: watch_flush enter 2024-03-29T10:51:20.749 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.767+0000 7fd294b3b640 10 librados: watch_flush exit 2024-03-29T10:51:20.749 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.767+0000 7fd294b3b640 20 client.4243.objecter shutdown clearing up homeless session... 2024-03-29T10:51:20.749 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.767+0000 7fd294b3b640 10 client.4243.objecter successfully canceled tick 2024-03-29T10:51:20.749 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.767+0000 7fd294b3b640 1 -- 172.21.15.112:0/1105902491 >> v1:172.21.15.118:6816/3181073408 conn(0x7fd27006de30 legacy=0x7fd2700702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:20.749 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.767+0000 7fd294b3b640 10 monclient: shutdown 2024-03-29T10:51:20.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.767+0000 7fd294b3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:20.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.767+0000 7fd294b3b640 1 -- 172.21.15.112:0/1105902491 >> v1:172.21.15.118:6789/0 conn(0x7fd290142690 legacy=0x7fd2901e6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:20.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.767+0000 7fd294b3b640 1 -- 172.21.15.112:0/1105902491 shutdown_connections 2024-03-29T10:51:20.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.767+0000 7fd294b3b640 1 -- 172.21.15.112:0/1105902491 >> 172.21.15.112:0/1105902491 conn(0x7fd290066a00 msgr2=0x7fd290066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:20.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.767+0000 7fd294b3b640 1 -- 172.21.15.112:0/1105902491 shutdown_connections 2024-03-29T10:51:20.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.768+0000 7fd294b3b640 1 -- 172.21.15.112:0/1105902491 wait complete. 2024-03-29T10:51:20.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:20.768+0000 7fd294b3b640 1 librados: shutdown 2024-03-29T10:51:20.764 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:27.571 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:27.713 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.731+0000 7f902992d640 10 monclient: get_monmap_and_config 2024-03-29T10:51:27.713 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.731+0000 7f902992d640 10 monclient: build_initial_monmap 2024-03-29T10:51:27.713 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.731+0000 7f902992d640 10 monclient: monmap: 2024-03-29T10:51:27.713 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:51:27.713 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:27.713 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:51:27.732757+0000 2024-03-29T10:51:27.713 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:51:27.732757+0000 2024-03-29T10:51:27.714 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:27.714 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:27.714 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:27.714 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:51:27.714 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:51:27.714 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:27.714 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.732+0000 7f902992d640 1 Processor -- start 2024-03-29T10:51:27.714 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.733+0000 7f902992d640 1 -- start start 2024-03-29T10:51:27.714 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.733+0000 7f902992d640 10 monclient: init 2024-03-29T10:51:27.717 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.734+0000 7f902992d640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:27.717 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.734+0000 7f902992d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:51:27.717 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.734+0000 7f902992d640 10 monclient(hunting): picked mon.noname-c con 0x7f901c02b000 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:27.717 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.734+0000 7f902992d640 10 monclient(hunting): picked mon.noname-a con 0x7f901c0c1d40 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:27.717 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.734+0000 7f902992d640 10 monclient(hunting): picked mon.noname-b con 0x7f901c0c4c00 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:27.717 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.734+0000 7f902992d640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f901c02b3e0 con 0x7f901c0c1d40 2024-03-29T10:51:27.718 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.734+0000 7f902912c640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f901c0c4c00 0x7f901c0cfdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:45074/0 (socket says 172.21.15.112:45074) 2024-03-29T10:51:27.718 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.734+0000 7f902912c640 1 -- 172.21.15.112:0/237170322 learned_addr learned my addr 172.21.15.112:0/237170322 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:51:27.718 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.734+0000 7f902992d640 1 -- 172.21.15.112:0/237170322 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f901c02c1f0 con 0x7f901c02b000 2024-03-29T10:51:27.718 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.734+0000 7f902992d640 1 -- 172.21.15.112:0/237170322 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f901c0c5050 con 0x7f901c0c4c00 2024-03-29T10:51:27.718 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.734+0000 7f902992d640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:27.718 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.735+0000 7f902992d640 10 monclient(hunting): authenticate will time out at 843.039180s 2024-03-29T10:51:27.718 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.735+0000 7f90237fe640 1 -- 172.21.15.112:0/237170322 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395818880 0 0) 0x7f901c02c1f0 con 0x7f901c02b000 2024-03-29T10:51:27.718 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.735+0000 7f90237fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:27.718 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.735+0000 7f90237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:27.718 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.735+0000 7f90237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:27.718 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.735+0000 7f90237fe640 10 monclient(hunting): my global_id is 4246 2024-03-29T10:51:27.718 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.735+0000 7f90237fe640 1 -- 172.21.15.112:0/237170322 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9014003520 con 0x7f901c02b000 2024-03-29T10:51:27.719 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.735+0000 7f90237fe640 1 -- 172.21.15.112:0/237170322 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685872982 0 0) 0x7f901c0c5050 con 0x7f901c0c4c00 2024-03-29T10:51:27.719 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.735+0000 7f90237fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:27.719 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.735+0000 7f90237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:27.719 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.735+0000 7f90237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:27.719 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.735+0000 7f90237fe640 10 monclient(hunting): my global_id is 4235 2024-03-29T10:51:27.719 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.735+0000 7f90237fe640 1 -- 172.21.15.112:0/237170322 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f901c02c1f0 con 0x7f901c0c4c00 2024-03-29T10:51:27.719 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.735+0000 7f90237fe640 1 -- 172.21.15.112:0/237170322 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1057002989 0 0) 0x7f9014003520 con 0x7f901c02b000 2024-03-29T10:51:27.719 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.735+0000 7f90237fe640 1 -- 172.21.15.112:0/237170322 >> v1:172.21.15.112:6790/0 conn(0x7f901c0c4c00 legacy=0x7f901c0cfdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:27.719 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.735+0000 7f90237fe640 1 -- 172.21.15.112:0/237170322 >> v1:172.21.15.112:6789/0 conn(0x7f901c0c1d40 legacy=0x7f901c0c44f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:51:27.722 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.737+0000 7f90237fe640 10 monclient: _finish_hunting 0 2024-03-29T10:51:27.722 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.737+0000 7f90237fe640 1 monclient: found mon.noname-c 2024-03-29T10:51:27.722 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.737+0000 7f90237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:27.722 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.737+0000 7f90237fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T10:51:27.722 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.737+0000 7f90237fe640 1 -- 172.21.15.112:0/237170322 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f901c02d530 con 0x7f901c02b000 2024-03-29T10:51:27.722 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.737+0000 7f90237fe640 10 monclient: _finish_auth 0 2024-03-29T10:51:27.723 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.737+0000 7f90237fe640 10 monclient: _check_auth_tickets 2024-03-29T10:51:27.723 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.737+0000 7f90237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:27.723 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.737+0000 7f90237fe640 1 -- 172.21.15.112:0/237170322 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9018001690 con 0x7f901c02b000 2024-03-29T10:51:27.723 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.737+0000 7f90237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:27.723 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.737+0000 7f90237fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:27.723 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.737+0000 7f90237fe640 10 monclient: dump: 2024-03-29T10:51:27.723 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:27.723 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:27.723 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:27.723 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:27.723 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:27.723 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:27.723 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:27.723 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:27.724 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:27.724 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:27.724 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.737+0000 7f902992d640 5 monclient: authenticate success, global_id 4246 2024-03-29T10:51:27.724 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.737+0000 7f902992d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:27.724 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.738+0000 7f90237fe640 1 -- 172.21.15.112:0/237170322 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9018001930 con 0x7f901c02b000 2024-03-29T10:51:27.724 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.738+0000 7f90237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:27.724 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.738+0000 7f90237fe640 1 -- 172.21.15.112:0/237170322 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9018001cc0 con 0x7f901c02b000 2024-03-29T10:51:27.724 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.738+0000 7f90237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:27.724 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.738+0000 7f90237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:27.724 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.738+0000 7f90237fe640 10 monclient: dump: 2024-03-29T10:51:27.724 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:27.724 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:27.724 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:27.725 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:27.725 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:27.725 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:27.725 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:27.725 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:27.725 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:27.725 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:27.725 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.738+0000 7f902992d640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:27.725 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.738+0000 7f902992d640 4 set_mon_vals no callback set 2024-03-29T10:51:27.725 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.739+0000 7f902992d640 10 monclient: shutdown 2024-03-29T10:51:27.725 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.739+0000 7f902992d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:27.725 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.739+0000 7f902992d640 1 -- 172.21.15.112:0/237170322 >> v1:172.21.15.118:6789/0 conn(0x7f901c02b000 legacy=0x7f901c0c1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:27.725 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.740+0000 7f902992d640 1 -- 172.21.15.112:0/237170322 shutdown_connections 2024-03-29T10:51:27.725 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.740+0000 7f902992d640 1 -- 172.21.15.112:0/237170322 >> 172.21.15.112:0/237170322 conn(0x7f901c03a590 msgr2=0x7f901c0367e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:27.725 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.740+0000 7f902992d640 1 -- 172.21.15.112:0/237170322 shutdown_connections 2024-03-29T10:51:27.733 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.740+0000 7f902992d640 1 -- 172.21.15.112:0/237170322 wait complete. 2024-03-29T10:51:27.733 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.742+0000 7f902992d640 10 monclient: build_initial_monmap 2024-03-29T10:51:27.733 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.742+0000 7f902992d640 10 monclient: monmap: 2024-03-29T10:51:27.733 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:51:27.733 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:27.733 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:51:27.733 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:51:27.733 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:27.733 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:27.734 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:27.734 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:51:27.734 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:51:27.734 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:27.734 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.743+0000 7f902992d640 1 librados: starting msgr at 2024-03-29T10:51:27.734 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.743+0000 7f902992d640 1 librados: starting objecter 2024-03-29T10:51:27.734 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.743+0000 7f902992d640 1 Processor -- start 2024-03-29T10:51:27.734 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.743+0000 7f902992d640 1 -- start start 2024-03-29T10:51:27.734 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.743+0000 7f902992d640 1 librados: setting wanted keys 2024-03-29T10:51:27.734 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.743+0000 7f902992d640 1 librados: calling monclient init 2024-03-29T10:51:27.734 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.743+0000 7f902992d640 10 monclient: init 2024-03-29T10:51:27.734 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.744+0000 7f902992d640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:27.734 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.744+0000 7f902992d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:27.734 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.745+0000 7f902992d640 10 monclient(hunting): picked mon.noname-c con 0x7f901c02b000 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:27.735 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.745+0000 7f902992d640 10 monclient(hunting): picked mon.noname-a con 0x7f901c0c1d40 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:27.735 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.745+0000 7f902992d640 10 monclient(hunting): picked mon.noname-b con 0x7f901c0c4c00 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:27.735 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.745+0000 7f902992d640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f901c0cec70 con 0x7f901c0c1d40 2024-03-29T10:51:27.735 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.745+0000 7f902992d640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f901c0ceec0 con 0x7f901c0c4c00 2024-03-29T10:51:27.735 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.745+0000 7f902992d640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f901c0d1f50 con 0x7f901c02b000 2024-03-29T10:51:27.735 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.745+0000 7f902992d640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:27.735 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.745+0000 7f902992d640 10 monclient(hunting): authenticate will time out at 843.049408s 2024-03-29T10:51:27.735 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.745+0000 7f902892b640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f901c02b000 0x7f901c0c2840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:45772/0 (socket says 172.21.15.112:45772) 2024-03-29T10:51:27.735 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.745+0000 7f902892b640 1 -- 172.21.15.112:0/4124075374 learned_addr learned my addr 172.21.15.112:0/4124075374 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:51:27.735 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.745+0000 7f90217fa640 10 client.?.objecter ms_handle_connect 0x7f901c02b000 2024-03-29T10:51:27.735 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.745+0000 7f90217fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:27.735 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.745+0000 7f90217fa640 1 -- 172.21.15.112:0/4124075374 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 486281266 0 0) 0x7f901c0d1f50 con 0x7f901c02b000 2024-03-29T10:51:27.735 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.745+0000 7f90217fa640 10 monclient(hunting): _negotiate 2024-03-29T10:51:27.735 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.745+0000 7f90217fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:27.736 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.745+0000 7f90217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:27.736 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.745+0000 7f90217fa640 10 monclient(hunting): my global_id is 4238 2024-03-29T10:51:27.736 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.745+0000 7f90217fa640 1 -- 172.21.15.112:0/4124075374 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9008004eb0 con 0x7f901c02b000 2024-03-29T10:51:27.736 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.746+0000 7f90217fa640 1 -- 172.21.15.112:0/4124075374 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2179649504 0 0) 0x7f9008004eb0 con 0x7f901c02b000 2024-03-29T10:51:27.736 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.746+0000 7f90217fa640 1 -- 172.21.15.112:0/4124075374 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9008005300 con 0x7f901c02b000 2024-03-29T10:51:27.736 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.746+0000 7f90217fa640 10 client.?.objecter ms_handle_connect 0x7f901c0c4c00 2024-03-29T10:51:27.736 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.746+0000 7f90217fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:27.736 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.746+0000 7f90217fa640 1 -- 172.21.15.112:0/4124075374 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9018001650 con 0x7f901c02b000 2024-03-29T10:51:27.736 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.746+0000 7f90217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:27.736 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.746+0000 7f90217fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:27.736 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.746+0000 7f90217fa640 10 monclient(hunting): dump: 2024-03-29T10:51:27.736 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:27.736 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:27.736 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:27.736 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:27.737 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:27.737 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:27.737 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:27.737 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:27.737 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:27.737 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:27.737 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.746+0000 7f90217fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:27.737 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f90217fa640 1 -- 172.21.15.112:0/4124075374 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2966544930 0 0) 0x7f9008005300 con 0x7f901c02b000 2024-03-29T10:51:27.737 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f90217fa640 1 -- 172.21.15.112:0/4124075374 >> v1:172.21.15.118:6789/0 conn(0x7f901c0c4c00 legacy=0x7f901c16b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:27.737 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f90217fa640 1 -- 172.21.15.112:0/4124075374 >> v1:172.21.15.112:6789/0 conn(0x7f901c0c1d40 legacy=0x7f901c0d29f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:51:27.737 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f90217fa640 10 monclient: _finish_hunting 0 2024-03-29T10:51:27.737 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f90217fa640 1 monclient: found mon.c 2024-03-29T10:51:27.737 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f90217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:27.737 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f90217fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:51:27.737 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f90217fa640 1 -- 172.21.15.112:0/4124075374 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f901c0d06d0 con 0x7f901c02b000 2024-03-29T10:51:27.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f90217fa640 10 monclient: _finish_auth 0 2024-03-29T10:51:27.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f90217fa640 10 monclient: _check_auth_tickets 2024-03-29T10:51:27.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f90217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:27.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f902992d640 5 monclient: authenticate success, global_id 4238 2024-03-29T10:51:27.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f902992d640 10 monclient: _renew_subs 2024-03-29T10:51:27.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f902992d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:51:27.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f902992d640 1 -- 172.21.15.112:0/4124075374 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f901c0c2f50 con 0x7f901c02b000 2024-03-29T10:51:27.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f902992d640 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:27.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f902992d640 10 monclient: _renew_subs 2024-03-29T10:51:27.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f902992d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:51:27.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f902992d640 1 -- 172.21.15.112:0/4124075374 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f901c0d32e0 con 0x7f901c02b000 2024-03-29T10:51:27.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f902992d640 1 librados: init done 2024-03-29T10:51:27.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f90217fa640 1 -- 172.21.15.112:0/4124075374 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9018003960 con 0x7f901c02b000 2024-03-29T10:51:27.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f90217fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:27.739 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f90217fa640 1 -- 172.21.15.112:0/4124075374 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9018001650 con 0x7f901c02b000 2024-03-29T10:51:27.739 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f90217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:27.739 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f90217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:27.739 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.747+0000 7f90217fa640 10 monclient: dump: 2024-03-29T10:51:27.739 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:27.739 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:27.739 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:27.739 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:27.739 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:27.739 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:27.739 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:27.739 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:27.739 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:27.740 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:27.740 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.748+0000 7f9022ffd640 4 set_mon_vals no callback set 2024-03-29T10:51:27.740 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.749+0000 7f900effd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:27.740 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.749+0000 7f900effd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:27.740 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.750+0000 7f900effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:51:27.740 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.750+0000 7f90217fa640 1 -- 172.21.15.112:0/4124075374 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f9018003c10 con 0x7f901c02b000 2024-03-29T10:51:27.740 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.750+0000 7f900effd640 1 -- 172.21.15.112:0/4124075374 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ffc005080 con 0x7f901c02b000 2024-03-29T10:51:27.740 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.751+0000 7f90217fa640 1 -- 172.21.15.112:0/4124075374 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f9018084970 con 0x7f901c02b000 2024-03-29T10:51:27.740 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.751+0000 7f90217fa640 10 client.4238.objecter ms_dispatch 0x7f9024000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:27.740 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.751+0000 7f90217fa640 3 client.4238.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:27.740 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.751+0000 7f90217fa640 3 client.4238.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:27.740 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.751+0000 7f90217fa640 20 client.4238.objecter dump_active .. 0 homeless 2024-03-29T10:51:27.740 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.751+0000 7f9023fff640 10 monclient: get_auth_request con 0x7f900806e1b0 auth_method 0 2024-03-29T10:51:27.741 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.752+0000 7f90217fa640 10 client.4238.objecter ms_handle_connect 0x7f900806e1b0 2024-03-29T10:51:27.741 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.758+0000 7f90217fa640 1 -- 172.21.15.112:0/4124075374 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9018050250 con 0x7f901c02b000 2024-03-29T10:51:27.741 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.758+0000 7f90217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:27.741 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:27.758+0000 7f90217fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:28.047 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:28.066+0000 7f900effd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:28.048 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:28.066+0000 7f900effd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:28.048 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:28.066+0000 7f900effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:51:28.048 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:28.066+0000 7f900effd640 1 -- 172.21.15.112:0/4124075374 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ffc005330 con 0x7f901c02b000 2024-03-29T10:51:28.048 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:28.067+0000 7f90217fa640 1 -- 172.21.15.112:0/4124075374 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f90180014b0 con 0x7f901c02b000 2024-03-29T10:51:28.049 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:28.067+0000 7f90217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:28.049 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:28.067+0000 7f90217fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:28.049 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:51:28.049 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:28.052 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:28.070+0000 7f900effd640 10 librados: watch_flush enter 2024-03-29T10:51:28.052 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:28.070+0000 7f900effd640 10 librados: watch_flush exit 2024-03-29T10:51:28.052 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:28.070+0000 7f900effd640 20 client.4238.objecter shutdown clearing up homeless session... 2024-03-29T10:51:28.052 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:28.070+0000 7f900effd640 10 client.4238.objecter successfully canceled tick 2024-03-29T10:51:28.052 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:28.070+0000 7f900effd640 1 -- 172.21.15.112:0/4124075374 >> v1:172.21.15.118:6816/3181073408 conn(0x7f900806e1b0 legacy=0x7f9008070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:28.053 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:28.070+0000 7f900effd640 10 monclient: shutdown 2024-03-29T10:51:28.053 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:28.070+0000 7f900effd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:28.053 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:28.070+0000 7f900effd640 1 -- 172.21.15.112:0/4124075374 >> v1:172.21.15.112:6790/0 conn(0x7f901c02b000 legacy=0x7f901c0c2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:28.053 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:28.070+0000 7f900effd640 1 -- 172.21.15.112:0/4124075374 shutdown_connections 2024-03-29T10:51:28.053 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:28.070+0000 7f900effd640 1 -- 172.21.15.112:0/4124075374 >> 172.21.15.112:0/4124075374 conn(0x7f901c03a590 msgr2=0x7f901c006490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:28.053 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:28.070+0000 7f900effd640 1 -- 172.21.15.112:0/4124075374 shutdown_connections 2024-03-29T10:51:28.053 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:28.070+0000 7f900effd640 1 -- 172.21.15.112:0/4124075374 wait complete. 2024-03-29T10:51:28.053 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:28.071+0000 7f900effd640 1 librados: shutdown 2024-03-29T10:51:28.064 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:34.870 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:35.014 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.032+0000 7fda294c5640 10 monclient: get_monmap_and_config 2024-03-29T10:51:35.014 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.032+0000 7fda294c5640 10 monclient: build_initial_monmap 2024-03-29T10:51:35.014 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.032+0000 7fda294c5640 10 monclient: monmap: 2024-03-29T10:51:35.014 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:51:35.014 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:35.014 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:51:35.033343+0000 2024-03-29T10:51:35.014 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:51:35.033343+0000 2024-03-29T10:51:35.014 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:35.014 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:35.014 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:35.014 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:51:35.015 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:51:35.015 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:35.015 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.033+0000 7fda294c5640 1 Processor -- start 2024-03-29T10:51:35.015 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.033+0000 7fda294c5640 1 -- start start 2024-03-29T10:51:35.015 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.033+0000 7fda294c5640 10 monclient: init 2024-03-29T10:51:35.016 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.034+0000 7fda294c5640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:35.016 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.034+0000 7fda294c5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:35.016 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.034+0000 7fda294c5640 10 monclient(hunting): picked mon.noname-b con 0x7fda2406ec30 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:35.016 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.034+0000 7fda294c5640 10 monclient(hunting): picked mon.noname-a con 0x7fda2406ff10 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:35.016 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.034+0000 7fda294c5640 10 monclient(hunting): picked mon.noname-c con 0x7fda24071f70 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:35.016 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.034+0000 7fda294c5640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda2406f5e0 con 0x7fda2406ff10 2024-03-29T10:51:35.016 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.034+0000 7fda294c5640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda24071440 con 0x7fda24071f70 2024-03-29T10:51:35.016 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.034+0000 7fda294c5640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda2413c6a0 con 0x7fda2406ec30 2024-03-29T10:51:35.016 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.034+0000 7fda294c5640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:35.016 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.034+0000 7fda294c5640 10 monclient(hunting): authenticate will time out at 850.338619s 2024-03-29T10:51:35.023 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.036+0000 7fda22ffd640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fda2406ec30 0x7fda2413bbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:38697/0 (socket says 172.21.15.112:38697) 2024-03-29T10:51:35.023 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.036+0000 7fda22ffd640 1 -- 172.21.15.112:0/2562220401 learned_addr learned my addr 172.21.15.112:0/2562220401 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:51:35.023 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.037+0000 7fda21ffb640 1 -- 172.21.15.112:0/2562220401 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1583530547 0 0) 0x7fda2413c6a0 con 0x7fda2406ec30 2024-03-29T10:51:35.023 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.037+0000 7fda21ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:35.023 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.037+0000 7fda21ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:35.023 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.037+0000 7fda21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:35.023 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.037+0000 7fda21ffb640 10 monclient(hunting): my global_id is 4241 2024-03-29T10:51:35.023 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.037+0000 7fda21ffb640 1 -- 172.21.15.112:0/2562220401 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda0c003520 con 0x7fda2406ec30 2024-03-29T10:51:35.023 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.037+0000 7fda21ffb640 1 -- 172.21.15.112:0/2562220401 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4253669488 0 0) 0x7fda0c003520 con 0x7fda2406ec30 2024-03-29T10:51:35.024 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.037+0000 7fda21ffb640 1 -- 172.21.15.112:0/2562220401 >> v1:172.21.15.118:6789/0 conn(0x7fda24071f70 legacy=0x7fda2414b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:35.024 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.037+0000 7fda21ffb640 1 -- 172.21.15.112:0/2562220401 >> v1:172.21.15.112:6789/0 conn(0x7fda2406ff10 legacy=0x7fda24071860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:51:35.024 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.039+0000 7fda21ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:51:35.024 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.039+0000 7fda21ffb640 1 monclient: found mon.noname-b 2024-03-29T10:51:35.024 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.039+0000 7fda21ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:35.024 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.039+0000 7fda21ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:51:35.024 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.039+0000 7fda21ffb640 1 -- 172.21.15.112:0/2562220401 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda24145460 con 0x7fda2406ec30 2024-03-29T10:51:35.024 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.039+0000 7fda21ffb640 10 monclient: _finish_auth 0 2024-03-29T10:51:35.024 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.039+0000 7fda21ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:51:35.024 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.039+0000 7fda21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:35.024 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.039+0000 7fda294c5640 5 monclient: authenticate success, global_id 4241 2024-03-29T10:51:35.024 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.039+0000 7fda294c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:35.024 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.039+0000 7fda21ffb640 1 -- 172.21.15.112:0/2562220401 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fda18001690 con 0x7fda2406ec30 2024-03-29T10:51:35.024 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.039+0000 7fda21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:35.024 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.039+0000 7fda21ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:35.025 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.039+0000 7fda21ffb640 10 monclient: dump: 2024-03-29T10:51:35.025 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:35.025 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:35.025 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:35.025 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:35.025 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:35.025 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:35.025 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:35.025 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:35.025 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:35.025 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:35.025 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.039+0000 7fda294c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:35.025 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.040+0000 7fda21ffb640 1 -- 172.21.15.112:0/2562220401 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda18001860 con 0x7fda2406ec30 2024-03-29T10:51:35.025 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.040+0000 7fda21ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:35.025 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.040+0000 7fda294c5640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:35.026 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.040+0000 7fda294c5640 4 set_mon_vals no callback set 2024-03-29T10:51:35.026 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.041+0000 7fda294c5640 10 monclient: shutdown 2024-03-29T10:51:35.026 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.041+0000 7fda294c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:35.026 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.041+0000 7fda294c5640 1 -- 172.21.15.112:0/2562220401 >> v1:172.21.15.112:6790/0 conn(0x7fda2406ec30 legacy=0x7fda2413bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:35.026 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.041+0000 7fda294c5640 1 -- 172.21.15.112:0/2562220401 shutdown_connections 2024-03-29T10:51:35.026 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.041+0000 7fda294c5640 1 -- 172.21.15.112:0/2562220401 >> 172.21.15.112:0/2562220401 conn(0x7fda240b48a0 msgr2=0x7fda240b4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:35.026 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.041+0000 7fda294c5640 1 -- 172.21.15.112:0/2562220401 shutdown_connections 2024-03-29T10:51:35.026 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.041+0000 7fda294c5640 1 -- 172.21.15.112:0/2562220401 wait complete. 2024-03-29T10:51:35.027 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.042+0000 7fda294c5640 10 monclient: build_initial_monmap 2024-03-29T10:51:35.027 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.042+0000 7fda294c5640 10 monclient: monmap: 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.042+0000 7fda294c5640 1 librados: starting msgr at 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.042+0000 7fda294c5640 1 librados: starting objecter 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.043+0000 7fda294c5640 1 Processor -- start 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.043+0000 7fda294c5640 1 -- start start 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.043+0000 7fda294c5640 1 librados: setting wanted keys 2024-03-29T10:51:35.029 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.043+0000 7fda294c5640 1 librados: calling monclient init 2024-03-29T10:51:35.029 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.043+0000 7fda294c5640 10 monclient: init 2024-03-29T10:51:35.029 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.044+0000 7fda294c5640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:35.029 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.044+0000 7fda294c5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:51:35.031 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.047+0000 7fda294c5640 10 monclient(hunting): picked mon.noname-a con 0x7fda2406ff10 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:35.031 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.047+0000 7fda294c5640 10 monclient(hunting): picked mon.noname-c con 0x7fda24071f70 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:35.031 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.047+0000 7fda294c5640 10 monclient(hunting): picked mon.noname-b con 0x7fda240b33c0 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:35.031 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.047+0000 7fda294c5640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda2413d4a0 con 0x7fda2406ff10 2024-03-29T10:51:35.031 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.047+0000 7fda294c5640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda2413d080 con 0x7fda240b33c0 2024-03-29T10:51:35.031 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.047+0000 7fda294c5640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda240b4620 con 0x7fda24071f70 2024-03-29T10:51:35.031 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.047+0000 7fda294c5640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:35.031 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.047+0000 7fda294c5640 10 monclient(hunting): authenticate will time out at 850.351847s 2024-03-29T10:51:35.031 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.048+0000 7fda237fe640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fda240b33c0 0x7fda2413e030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:58062/0 (socket says 172.21.15.112:58062) 2024-03-29T10:51:35.031 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.048+0000 7fda237fe640 1 -- 172.21.15.112:0/163722048 learned_addr learned my addr 172.21.15.112:0/163722048 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:51:35.031 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.048+0000 7fda0b7fe640 10 client.?.objecter ms_handle_connect 0x7fda240b33c0 2024-03-29T10:51:35.031 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.048+0000 7fda0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:35.031 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.048+0000 7fda0b7fe640 10 client.?.objecter ms_handle_connect 0x7fda2406ff10 2024-03-29T10:51:35.031 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.048+0000 7fda0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:35.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.048+0000 7fda0b7fe640 1 -- 172.21.15.112:0/163722048 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1145214466 0 0) 0x7fda2413d4a0 con 0x7fda2406ff10 2024-03-29T10:51:35.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.048+0000 7fda0b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:35.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.048+0000 7fda0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:35.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.048+0000 7fda0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:35.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.048+0000 7fda0b7fe640 10 monclient(hunting): my global_id is 4236 2024-03-29T10:51:35.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.049+0000 7fda0b7fe640 1 -- 172.21.15.112:0/163722048 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda10004eb0 con 0x7fda2406ff10 2024-03-29T10:51:35.037 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.050+0000 7fda0b7fe640 10 client.?.objecter ms_handle_connect 0x7fda24071f70 2024-03-29T10:51:35.037 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.050+0000 7fda0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:35.037 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.050+0000 7fda0b7fe640 1 -- 172.21.15.112:0/163722048 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724867925 0 0) 0x7fda240b4620 con 0x7fda24071f70 2024-03-29T10:51:35.037 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.050+0000 7fda0b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.050+0000 7fda0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.050+0000 7fda0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.050+0000 7fda0b7fe640 10 monclient(hunting): my global_id is 4244 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.051+0000 7fda0b7fe640 1 -- 172.21.15.112:0/163722048 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda2413d4a0 con 0x7fda24071f70 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.051+0000 7fda0b7fe640 1 -- 172.21.15.112:0/163722048 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 933331661 0 0) 0x7fda2413d080 con 0x7fda240b33c0 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.051+0000 7fda0b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.051+0000 7fda0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.051+0000 7fda0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.051+0000 7fda0b7fe640 10 monclient(hunting): my global_id is 4252 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.051+0000 7fda0b7fe640 1 -- 172.21.15.112:0/163722048 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda240b4620 con 0x7fda240b33c0 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.051+0000 7fda0b7fe640 1 -- 172.21.15.112:0/163722048 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3457805963 0 0) 0x7fda10004eb0 con 0x7fda2406ff10 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.051+0000 7fda0b7fe640 1 -- 172.21.15.112:0/163722048 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda2413d080 con 0x7fda2406ff10 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.051+0000 7fda0b7fe640 1 -- 172.21.15.112:0/163722048 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fda18004940 con 0x7fda2406ff10 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.051+0000 7fda0b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.052+0000 7fda0b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.052+0000 7fda0b7fe640 10 monclient(hunting): dump: 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.052+0000 7fda0b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.052+0000 7fda0b7fe640 1 -- 172.21.15.112:0/163722048 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1119986712 0 0) 0x7fda240b4620 con 0x7fda240b33c0 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.052+0000 7fda0b7fe640 1 -- 172.21.15.112:0/163722048 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda10004eb0 con 0x7fda240b33c0 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.052+0000 7fda0b7fe640 1 -- 172.21.15.112:0/163722048 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 502277149 0 0) 0x7fda2413d080 con 0x7fda2406ff10 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.052+0000 7fda0b7fe640 1 -- 172.21.15.112:0/163722048 >> v1:172.21.15.112:6790/0 conn(0x7fda24071f70 legacy=0x7fda2413d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.052+0000 7fda0b7fe640 1 -- 172.21.15.112:0/163722048 >> v1:172.21.15.118:6789/0 conn(0x7fda240b33c0 legacy=0x7fda2413e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.052+0000 7fda0b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.052+0000 7fda0b7fe640 1 monclient: found mon.a 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.052+0000 7fda0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.052+0000 7fda0b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.052+0000 7fda0b7fe640 1 -- 172.21.15.112:0/163722048 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda24144710 con 0x7fda2406ff10 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.052+0000 7fda0b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.052+0000 7fda0b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.053+0000 7fda0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.053+0000 7fda294c5640 5 monclient: authenticate success, global_id 4236 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.053+0000 7fda294c5640 10 monclient: _renew_subs 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.053+0000 7fda294c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:51:35.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.053+0000 7fda294c5640 1 -- 172.21.15.112:0/163722048 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda2412a230 con 0x7fda2406ff10 2024-03-29T10:51:35.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.053+0000 7fda294c5640 10 client.4236.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:35.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.053+0000 7fda294c5640 10 monclient: _renew_subs 2024-03-29T10:51:35.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.053+0000 7fda294c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:51:35.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.053+0000 7fda294c5640 1 -- 172.21.15.112:0/163722048 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda241e7180 con 0x7fda2406ff10 2024-03-29T10:51:35.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.053+0000 7fda294c5640 1 librados: init done 2024-03-29T10:51:35.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.053+0000 7fda0b7fe640 1 -- 172.21.15.112:0/163722048 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda18001b10 con 0x7fda2406ff10 2024-03-29T10:51:35.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.053+0000 7fda0b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:35.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.053+0000 7fda0b7fe640 1 -- 172.21.15.112:0/163722048 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fda18004f70 con 0x7fda2406ff10 2024-03-29T10:51:35.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.053+0000 7fda0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:35.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.053+0000 7fda0b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:35.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.053+0000 7fda0b7fe640 10 monclient: dump: 2024-03-29T10:51:35.041 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:35.041 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:35.041 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:35.042 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:35.042 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:35.042 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:35.042 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:35.042 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:35.042 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:35.042 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:35.042 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.054+0000 7fda20ff9640 4 set_mon_vals no callback set 2024-03-29T10:51:35.042 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.054+0000 7fda294c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:35.042 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.055+0000 7fda294c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:35.042 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.055+0000 7fda294c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:51:35.042 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.055+0000 7fda294c5640 1 -- 172.21.15.112:0/163722048 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda04005080 con 0x7fda2406ff10 2024-03-29T10:51:35.042 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.056+0000 7fda0b7fe640 1 -- 172.21.15.112:0/163722048 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fda1801af40 con 0x7fda2406ff10 2024-03-29T10:51:35.043 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.056+0000 7fda0b7fe640 1 -- 172.21.15.112:0/163722048 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7fda18084bd0 con 0x7fda2406ff10 2024-03-29T10:51:35.043 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.056+0000 7fda0b7fe640 10 client.4236.objecter ms_dispatch 0x7fda24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:35.043 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.057+0000 7fda0b7fe640 3 client.4236.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:35.043 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.057+0000 7fda0b7fe640 3 client.4236.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:35.043 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.057+0000 7fda227fc640 10 monclient: get_auth_request con 0x7fda1006dec0 auth_method 0 2024-03-29T10:51:35.043 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.057+0000 7fda0b7fe640 20 client.4236.objecter dump_active .. 0 homeless 2024-03-29T10:51:35.043 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.058+0000 7fda0b7fe640 10 client.4236.objecter ms_handle_connect 0x7fda1006dec0 2024-03-29T10:51:35.046 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.064+0000 7fda0b7fe640 1 -- 172.21.15.112:0/163722048 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fda180504b0 con 0x7fda2406ff10 2024-03-29T10:51:35.046 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.065+0000 7fda0b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:35.046 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.065+0000 7fda0b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:35.365 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.383+0000 7fda294c5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:35.365 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.383+0000 7fda294c5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:35.365 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.384+0000 7fda294c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:51:35.365 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.384+0000 7fda294c5640 1 -- 172.21.15.112:0/163722048 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda04005330 con 0x7fda2406ff10 2024-03-29T10:51:35.366 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.385+0000 7fda0b7fe640 1 -- 172.21.15.112:0/163722048 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7fda180523e0 con 0x7fda2406ff10 2024-03-29T10:51:35.366 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.385+0000 7fda0b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:35.366 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.385+0000 7fda0b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:35.366 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:51:35.367 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:35.371 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.388+0000 7fda294c5640 10 librados: watch_flush enter 2024-03-29T10:51:35.371 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.388+0000 7fda294c5640 10 librados: watch_flush exit 2024-03-29T10:51:35.371 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.388+0000 7fda294c5640 20 client.4236.objecter shutdown clearing up homeless session... 2024-03-29T10:51:35.371 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.388+0000 7fda294c5640 10 client.4236.objecter successfully canceled tick 2024-03-29T10:51:35.371 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.389+0000 7fda294c5640 1 -- 172.21.15.112:0/163722048 >> v1:172.21.15.118:6816/3181073408 conn(0x7fda1006dec0 legacy=0x7fda10070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:35.371 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.389+0000 7fda294c5640 10 monclient: shutdown 2024-03-29T10:51:35.371 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.389+0000 7fda294c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:35.371 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.389+0000 7fda294c5640 1 -- 172.21.15.112:0/163722048 >> v1:172.21.15.112:6789/0 conn(0x7fda2406ff10 legacy=0x7fda240b2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:35.372 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.389+0000 7fda294c5640 1 -- 172.21.15.112:0/163722048 shutdown_connections 2024-03-29T10:51:35.372 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.389+0000 7fda294c5640 1 -- 172.21.15.112:0/163722048 >> 172.21.15.112:0/163722048 conn(0x7fda240b48a0 msgr2=0x7fda24069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:35.372 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.390+0000 7fda294c5640 1 -- 172.21.15.112:0/163722048 shutdown_connections 2024-03-29T10:51:35.372 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.390+0000 7fda294c5640 1 -- 172.21.15.112:0/163722048 wait complete. 2024-03-29T10:51:35.372 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:35.391+0000 7fda294c5640 1 librados: shutdown 2024-03-29T10:51:35.390 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:42.197 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:42.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.370+0000 7f5639e55640 10 monclient: get_monmap_and_config 2024-03-29T10:51:42.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.370+0000 7f5639e55640 10 monclient: build_initial_monmap 2024-03-29T10:51:42.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.370+0000 7f5639e55640 10 monclient: monmap: 2024-03-29T10:51:42.353 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:51:42.353 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:42.353 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:51:42.371120+0000 2024-03-29T10:51:42.353 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:51:42.371120+0000 2024-03-29T10:51:42.353 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:42.354 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:42.354 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:42.354 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:51:42.354 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:51:42.354 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:42.357 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.375+0000 7f5639e55640 1 Processor -- start 2024-03-29T10:51:42.357 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.375+0000 7f5639e55640 1 -- start start 2024-03-29T10:51:42.357 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.375+0000 7f5639e55640 10 monclient: init 2024-03-29T10:51:42.358 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.376+0000 7f5639e55640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:42.358 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.376+0000 7f5639e55640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:51:42.358 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.376+0000 7f5639e55640 10 monclient(hunting): picked mon.noname-c con 0x7f56340aae30 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:42.358 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.376+0000 7f5639e55640 10 monclient(hunting): picked mon.noname-b con 0x7f56340a9e70 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:42.358 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.376+0000 7f5639e55640 10 monclient(hunting): picked mon.noname-a con 0x7f5634146290 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:42.358 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.376+0000 7f5639e55640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56340ac580 con 0x7f5634146290 2024-03-29T10:51:42.358 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.376+0000 7f5639e55640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56340ac9a0 con 0x7f56340aae30 2024-03-29T10:51:42.358 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.376+0000 7f5639e55640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56340ad640 con 0x7f56340a9e70 2024-03-29T10:51:42.358 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.376+0000 7f5639e55640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:42.358 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.376+0000 7f5639e55640 10 monclient(hunting): authenticate will time out at 857.680862s 2024-03-29T10:51:42.361 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.380+0000 7f5632ffd640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f56340a9e70 0x7f56340acf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35357/0 (socket says 172.21.15.112:35357) 2024-03-29T10:51:42.367 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.380+0000 7f5632ffd640 1 -- 172.21.15.112:0/394832986 learned_addr learned my addr 172.21.15.112:0/394832986 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:51:42.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.380+0000 7f56327fc640 1 -- 172.21.15.112:0/394832986 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197353375 0 0) 0x7f56340ac580 con 0x7f5634146290 2024-03-29T10:51:42.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.380+0000 7f56327fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:42.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.380+0000 7f56327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:42.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.380+0000 7f56327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:42.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.380+0000 7f56327fc640 10 monclient(hunting): my global_id is 4239 2024-03-29T10:51:42.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.380+0000 7f56327fc640 1 -- 172.21.15.112:0/394832986 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5624003520 con 0x7f5634146290 2024-03-29T10:51:42.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.381+0000 7f56327fc640 1 -- 172.21.15.112:0/394832986 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 152301131 0 0) 0x7f5624003520 con 0x7f5634146290 2024-03-29T10:51:42.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.381+0000 7f56327fc640 1 -- 172.21.15.112:0/394832986 >> v1:172.21.15.112:6790/0 conn(0x7f56340a9e70 legacy=0x7f56340acf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:51:42.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.381+0000 7f56327fc640 1 -- 172.21.15.112:0/394832986 >> v1:172.21.15.118:6789/0 conn(0x7f56340aae30 legacy=0x7f5634146aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:51:42.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.382+0000 7f56327fc640 10 monclient: _finish_hunting 0 2024-03-29T10:51:42.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.382+0000 7f56327fc640 1 monclient: found mon.noname-a 2024-03-29T10:51:42.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.382+0000 7f56327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:42.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.382+0000 7f56327fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T10:51:42.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.382+0000 7f56327fc640 1 -- 172.21.15.112:0/394832986 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56340ade20 con 0x7f5634146290 2024-03-29T10:51:42.369 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.382+0000 7f56327fc640 10 monclient: _finish_auth 0 2024-03-29T10:51:42.369 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.382+0000 7f56327fc640 10 monclient: _check_auth_tickets 2024-03-29T10:51:42.369 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.382+0000 7f56327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:42.369 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.382+0000 7f56327fc640 1 -- 172.21.15.112:0/394832986 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f562c001a40 con 0x7f5634146290 2024-03-29T10:51:42.369 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.382+0000 7f56327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:42.369 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.382+0000 7f56327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:42.369 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.382+0000 7f56327fc640 10 monclient: dump: 2024-03-29T10:51:42.369 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:42.369 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:42.369 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:42.369 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:42.369 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:42.369 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:42.369 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:42.370 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:42.370 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:42.370 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:42.370 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.383+0000 7f5639e55640 5 monclient: authenticate success, global_id 4239 2024-03-29T10:51:42.370 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.383+0000 7f5639e55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:42.370 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.385+0000 7f56327fc640 1 -- 172.21.15.112:0/394832986 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f562c001c10 con 0x7f5634146290 2024-03-29T10:51:42.370 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.385+0000 7f56327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:42.370 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.385+0000 7f56327fc640 1 -- 172.21.15.112:0/394832986 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f562c002060 con 0x7f5634146290 2024-03-29T10:51:42.370 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.385+0000 7f56327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:42.370 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.385+0000 7f56327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:42.370 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.385+0000 7f56327fc640 10 monclient: dump: 2024-03-29T10:51:42.370 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:42.370 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:42.370 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:42.371 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:42.371 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:42.371 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:42.371 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:42.371 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:42.371 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:42.371 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:42.371 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.385+0000 7f5639e55640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:42.371 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.385+0000 7f5639e55640 4 set_mon_vals no callback set 2024-03-29T10:51:42.371 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.386+0000 7f5639e55640 10 monclient: shutdown 2024-03-29T10:51:42.371 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.386+0000 7f5639e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:42.372 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.386+0000 7f5639e55640 1 -- 172.21.15.112:0/394832986 >> v1:172.21.15.112:6789/0 conn(0x7f5634146290 legacy=0x7f5634151ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:42.372 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.386+0000 7f5639e55640 1 -- 172.21.15.112:0/394832986 shutdown_connections 2024-03-29T10:51:42.372 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.386+0000 7f5639e55640 1 -- 172.21.15.112:0/394832986 >> 172.21.15.112:0/394832986 conn(0x7f56340b3c80 msgr2=0x7f56340b4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:42.372 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.386+0000 7f5639e55640 1 -- 172.21.15.112:0/394832986 shutdown_connections 2024-03-29T10:51:42.372 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.386+0000 7f5639e55640 1 -- 172.21.15.112:0/394832986 wait complete. 2024-03-29T10:51:42.372 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.387+0000 7f5639e55640 10 monclient: build_initial_monmap 2024-03-29T10:51:42.372 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.387+0000 7f5639e55640 10 monclient: monmap: 2024-03-29T10:51:42.373 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:51:42.373 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:42.373 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:51:42.373 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:51:42.373 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:42.373 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:42.373 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:42.373 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:51:42.373 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:51:42.373 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:42.373 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.388+0000 7f5639e55640 1 librados: starting msgr at 2024-03-29T10:51:42.373 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.388+0000 7f5639e55640 1 librados: starting objecter 2024-03-29T10:51:42.373 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.388+0000 7f5639e55640 1 Processor -- start 2024-03-29T10:51:42.373 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.388+0000 7f5639e55640 1 -- start start 2024-03-29T10:51:42.374 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.388+0000 7f5639e55640 1 librados: setting wanted keys 2024-03-29T10:51:42.374 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.388+0000 7f5639e55640 1 librados: calling monclient init 2024-03-29T10:51:42.374 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.388+0000 7f5639e55640 10 monclient: init 2024-03-29T10:51:42.376 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.389+0000 7f5639e55640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:42.376 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.389+0000 7f5639e55640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:51:42.376 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.392+0000 7f56337fe640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f56340aae30 0x7f56341d8470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35415/0 (socket says 172.21.15.112:35415) 2024-03-29T10:51:42.376 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.392+0000 7f5639e55640 10 monclient(hunting): picked mon.noname-c con 0x7f56340aae30 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:42.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.395+0000 7f5639e55640 10 monclient(hunting): picked mon.noname-b con 0x7f5634146290 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:42.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.395+0000 7f5639e55640 10 monclient(hunting): picked mon.noname-a con 0x7f5634149470 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:42.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.395+0000 7f5639e55640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56341d8f80 con 0x7f5634149470 2024-03-29T10:51:42.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.395+0000 7f5639e55640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56341d8b80 con 0x7f5634146290 2024-03-29T10:51:42.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.395+0000 7f5639e55640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56341d8d10 con 0x7f56340aae30 2024-03-29T10:51:42.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.395+0000 7f5639e55640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:42.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.395+0000 7f5639e55640 10 monclient(hunting): authenticate will time out at 857.699980s 2024-03-29T10:51:42.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.395+0000 7f5632ffd640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f5634146290 0x7f5634137300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:49627/0 (socket says 172.21.15.112:49627) 2024-03-29T10:51:42.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.395+0000 7f5632ffd640 1 -- 172.21.15.112:0/1686607323 learned_addr learned my addr 172.21.15.112:0/1686607323 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:51:42.383 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.396+0000 7f561ffff640 10 client.?.objecter ms_handle_connect 0x7f5634146290 2024-03-29T10:51:42.383 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.396+0000 7f561ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:42.383 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.396+0000 7f561ffff640 10 client.?.objecter ms_handle_connect 0x7f56340aae30 2024-03-29T10:51:42.383 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.396+0000 7f561ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:42.383 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.396+0000 7f561ffff640 1 -- 172.21.15.112:0/1686607323 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3361992503 0 0) 0x7f56341d8b80 con 0x7f5634146290 2024-03-29T10:51:42.383 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.396+0000 7f561ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:42.384 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.396+0000 7f561ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:42.384 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.396+0000 7f561ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:42.384 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.396+0000 7f561ffff640 10 monclient(hunting): my global_id is 4255 2024-03-29T10:51:42.384 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.396+0000 7f561ffff640 1 -- 172.21.15.112:0/1686607323 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5618004eb0 con 0x7f5634146290 2024-03-29T10:51:42.384 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.396+0000 7f561ffff640 1 -- 172.21.15.112:0/1686607323 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 166466566 0 0) 0x7f5618004eb0 con 0x7f5634146290 2024-03-29T10:51:42.384 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.396+0000 7f561ffff640 1 -- 172.21.15.112:0/1686607323 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56341d8b80 con 0x7f5634146290 2024-03-29T10:51:42.384 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.396+0000 7f561ffff640 1 -- 172.21.15.112:0/1686607323 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f5628001510 con 0x7f5634146290 2024-03-29T10:51:42.384 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.396+0000 7f561ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:42.384 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.396+0000 7f561ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:42.384 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.396+0000 7f561ffff640 10 monclient(hunting): dump: 2024-03-29T10:51:42.384 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:42.384 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:42.384 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:42.384 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.396+0000 7f561ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.396+0000 7f561ffff640 1 -- 172.21.15.112:0/1686607323 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128460817 0 0) 0x7f56341d8d10 con 0x7f56340aae30 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.396+0000 7f561ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.396+0000 7f561ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.396+0000 7f561ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.396+0000 7f561ffff640 10 monclient(hunting): my global_id is 4247 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.396+0000 7f561ffff640 1 -- 172.21.15.112:0/1686607323 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5618004eb0 con 0x7f56340aae30 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.397+0000 7f561ffff640 1 -- 172.21.15.112:0/1686607323 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 963325944 0 0) 0x7f56341d8b80 con 0x7f5634146290 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.397+0000 7f561ffff640 1 -- 172.21.15.112:0/1686607323 >> v1:172.21.15.112:6790/0 conn(0x7f56340aae30 legacy=0x7f56341d8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:42.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.397+0000 7f561ffff640 1 -- 172.21.15.112:0/1686607323 >> v1:172.21.15.112:6789/0 conn(0x7f5634149470 legacy=0x7f56341db680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:51:42.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.397+0000 7f561ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:51:42.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.397+0000 7f561ffff640 1 monclient: found mon.b 2024-03-29T10:51:42.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.397+0000 7f561ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:42.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.397+0000 7f561ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:42.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.397+0000 7f561ffff640 1 -- 172.21.15.112:0/1686607323 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5634137a10 con 0x7f5634146290 2024-03-29T10:51:42.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.397+0000 7f561ffff640 10 monclient: _finish_auth 0 2024-03-29T10:51:42.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.397+0000 7f561ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:51:42.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.397+0000 7f561ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:42.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.398+0000 7f5639e55640 5 monclient: authenticate success, global_id 4255 2024-03-29T10:51:42.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.398+0000 7f5639e55640 10 monclient: _renew_subs 2024-03-29T10:51:42.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.398+0000 7f5639e55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:42.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.398+0000 7f5639e55640 1 -- 172.21.15.112:0/1686607323 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5634137c60 con 0x7f5634146290 2024-03-29T10:51:42.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.398+0000 7f5639e55640 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.398+0000 7f5639e55640 10 monclient: _renew_subs 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.398+0000 7f5639e55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.398+0000 7f5639e55640 1 -- 172.21.15.112:0/1686607323 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f56341dbf70 con 0x7f5634146290 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.398+0000 7f5639e55640 1 librados: init done 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.398+0000 7f561ffff640 1 -- 172.21.15.112:0/1686607323 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5628001aa0 con 0x7f5634146290 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.398+0000 7f561ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.398+0000 7f561ffff640 1 -- 172.21.15.112:0/1686607323 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f5628004ba0 con 0x7f5634146290 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.398+0000 7f561ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.398+0000 7f561ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.398+0000 7f561ffff640 10 monclient: dump: 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:42.388 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:42.388 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:42.388 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:42.388 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:42.388 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:42.388 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:42.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.399+0000 7f5631ffb640 4 set_mon_vals no callback set 2024-03-29T10:51:42.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.400+0000 7f561ffff640 1 -- 172.21.15.112:0/1686607323 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f562801ab70 con 0x7f5634146290 2024-03-29T10:51:42.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.401+0000 7f561ffff640 1 -- 172.21.15.112:0/1686607323 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f5628084b10 con 0x7f5634146290 2024-03-29T10:51:42.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.401+0000 7f561ffff640 10 client.4255.objecter ms_dispatch 0x7f5634000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:42.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.401+0000 7f561ffff640 3 client.4255.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:42.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.401+0000 7f561ffff640 3 client.4255.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:42.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.401+0000 7f561ffff640 20 client.4255.objecter dump_active .. 0 homeless 2024-03-29T10:51:42.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.401+0000 7f5639e55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:42.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.401+0000 7f5639e55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:42.389 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.401+0000 7f5639e55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:42.389 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.401+0000 7f5639e55640 1 -- 172.21.15.112:0/1686607323 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5614005080 con 0x7f5634146290 2024-03-29T10:51:42.389 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.402+0000 7f56337fe640 10 monclient: get_auth_request con 0x7f561806dd90 auth_method 0 2024-03-29T10:51:42.389 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.402+0000 7f561ffff640 10 client.4255.objecter ms_handle_connect 0x7f561806dd90 2024-03-29T10:51:42.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.412+0000 7f561ffff640 1 -- 172.21.15.112:0/1686607323 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5628050220 con 0x7f5634146290 2024-03-29T10:51:42.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.412+0000 7f561ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:42.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.412+0000 7f561ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:42.720 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.738+0000 7f5639e55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:42.720 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.738+0000 7f5639e55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:42.720 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.738+0000 7f5639e55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:51:42.720 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.739+0000 7f5639e55640 1 -- 172.21.15.112:0/1686607323 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5614005330 con 0x7f5634146290 2024-03-29T10:51:42.721 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.740+0000 7f561ffff640 1 -- 172.21.15.112:0/1686607323 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f5628052320 con 0x7f5634146290 2024-03-29T10:51:42.721 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.740+0000 7f561ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:42.721 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.740+0000 7f561ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:42.723 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:51:42.723 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:42.726 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.745+0000 7f5639e55640 10 librados: watch_flush enter 2024-03-29T10:51:42.726 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.745+0000 7f5639e55640 10 librados: watch_flush exit 2024-03-29T10:51:42.726 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.745+0000 7f5639e55640 20 client.4255.objecter shutdown clearing up homeless session... 2024-03-29T10:51:42.727 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.745+0000 7f5639e55640 10 client.4255.objecter successfully canceled tick 2024-03-29T10:51:42.728 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.747+0000 7f5639e55640 1 -- 172.21.15.112:0/1686607323 >> v1:172.21.15.118:6816/3181073408 conn(0x7f561806dd90 legacy=0x7f5618070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:42.728 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.747+0000 7f5639e55640 10 monclient: shutdown 2024-03-29T10:51:42.728 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.747+0000 7f5639e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:42.728 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.747+0000 7f5639e55640 1 -- 172.21.15.112:0/1686607323 >> v1:172.21.15.118:6789/0 conn(0x7f5634146290 legacy=0x7f5634137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:42.729 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.748+0000 7f5639e55640 1 -- 172.21.15.112:0/1686607323 shutdown_connections 2024-03-29T10:51:42.729 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.748+0000 7f5639e55640 1 -- 172.21.15.112:0/1686607323 >> 172.21.15.112:0/1686607323 conn(0x7f56340b3c80 msgr2=0x7f5634146870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:42.729 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.748+0000 7f5639e55640 1 -- 172.21.15.112:0/1686607323 shutdown_connections 2024-03-29T10:51:42.729 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.748+0000 7f5639e55640 1 -- 172.21.15.112:0/1686607323 wait complete. 2024-03-29T10:51:42.730 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:42.748+0000 7f5639e55640 1 librados: shutdown 2024-03-29T10:51:42.748 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:49.556 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:49.714 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.732+0000 7f28ae1f6640 10 monclient: get_monmap_and_config 2024-03-29T10:51:49.714 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.732+0000 7f28ae1f6640 10 monclient: build_initial_monmap 2024-03-29T10:51:49.714 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.732+0000 7f28ae1f6640 10 monclient: monmap: 2024-03-29T10:51:49.714 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:51:49.714 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:49.714 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:51:49.733588+0000 2024-03-29T10:51:49.714 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:51:49.733588+0000 2024-03-29T10:51:49.714 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:49.714 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:49.715 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:49.715 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:51:49.715 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:51:49.715 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:49.716 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.735+0000 7f28ae1f6640 1 Processor -- start 2024-03-29T10:51:49.717 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.735+0000 7f28ae1f6640 1 -- start start 2024-03-29T10:51:49.717 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.735+0000 7f28ae1f6640 10 monclient: init 2024-03-29T10:51:49.717 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.736+0000 7f28ae1f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:49.718 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.736+0000 7f28ae1f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:51:49.718 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.736+0000 7f28ae1f6640 10 monclient(hunting): picked mon.noname-b con 0x7f28a81457f0 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:49.718 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.736+0000 7f28ae1f6640 10 monclient(hunting): picked mon.noname-c con 0x7f28a813f650 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:49.718 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.736+0000 7f28ae1f6640 10 monclient(hunting): picked mon.noname-a con 0x7f28a80a9d70 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:49.718 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.736+0000 7f28ae1f6640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a80aaef0 con 0x7f28a80a9d70 2024-03-29T10:51:49.718 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.736+0000 7f28ae1f6640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a8145bd0 con 0x7f28a813f650 2024-03-29T10:51:49.718 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.736+0000 7f28ae1f6640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a8145eb0 con 0x7f28a81457f0 2024-03-29T10:51:49.718 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.736+0000 7f28ae1f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:49.718 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.736+0000 7f28ae1f6640 10 monclient(hunting): authenticate will time out at 865.040849s 2024-03-29T10:51:49.721 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.739+0000 7f28a7fff640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f28a80a9d70 0x7f28a814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:52169/0 (socket says 172.21.15.112:52169) 2024-03-29T10:51:49.721 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.739+0000 7f28a7fff640 1 -- 172.21.15.112:0/4015377083 learned_addr learned my addr 172.21.15.112:0/4015377083 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:51:49.725 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.740+0000 7f28a67fc640 1 -- 172.21.15.112:0/4015377083 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811094703 0 0) 0x7f28a80aaef0 con 0x7f28a80a9d70 2024-03-29T10:51:49.725 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.740+0000 7f28a67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:49.725 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.740+0000 7f28a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:49.725 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.740+0000 7f28a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:49.726 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.740+0000 7f28a67fc640 10 monclient(hunting): my global_id is 4242 2024-03-29T10:51:49.726 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.740+0000 7f28a67fc640 1 -- 172.21.15.112:0/4015377083 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2890003520 con 0x7f28a80a9d70 2024-03-29T10:51:49.726 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 1 -- 172.21.15.112:0/4015377083 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318046562 0 0) 0x7f28a8145bd0 con 0x7f28a813f650 2024-03-29T10:51:49.726 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:49.726 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:49.726 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:49.726 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 10 monclient(hunting): my global_id is 4258 2024-03-29T10:51:49.726 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 1 -- 172.21.15.112:0/4015377083 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28a80aaef0 con 0x7f28a813f650 2024-03-29T10:51:49.726 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 1 -- 172.21.15.112:0/4015377083 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3811528582 0 0) 0x7f28a80aaef0 con 0x7f28a813f650 2024-03-29T10:51:49.726 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 1 -- 172.21.15.112:0/4015377083 >> v1:172.21.15.112:6790/0 conn(0x7f28a81457f0 legacy=0x7f28a813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:51:49.726 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 1 -- 172.21.15.112:0/4015377083 >> v1:172.21.15.112:6789/0 conn(0x7f28a80a9d70 legacy=0x7f28a814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:49.727 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:51:49.730 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 1 monclient: found mon.noname-c 2024-03-29T10:51:49.730 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:49.730 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T10:51:49.730 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 1 -- 172.21.15.112:0/4015377083 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a8146340 con 0x7f28a813f650 2024-03-29T10:51:49.730 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 10 monclient: _finish_auth 0 2024-03-29T10:51:49.730 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:51:49.730 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:49.730 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 1 -- 172.21.15.112:0/4015377083 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f2898001670 con 0x7f28a813f650 2024-03-29T10:51:49.730 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:49.730 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:49.730 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 10 monclient: dump: 2024-03-29T10:51:49.730 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:49.730 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:49.731 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:49.731 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:49.731 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:49.731 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:49.731 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:49.731 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:49.731 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:49.731 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:49.731 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28ae1f6640 5 monclient: authenticate success, global_id 4258 2024-03-29T10:51:49.731 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28ae1f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:49.731 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 1 -- 172.21.15.112:0/4015377083 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28980017d0 con 0x7f28a813f650 2024-03-29T10:51:49.731 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:49.731 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 1 -- 172.21.15.112:0/4015377083 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f2898001be0 con 0x7f28a813f650 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28a67fc640 10 monclient: dump: 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28ae1f6640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.743+0000 7f28ae1f6640 4 set_mon_vals no callback set 2024-03-29T10:51:49.733 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.744+0000 7f28ae1f6640 10 monclient: shutdown 2024-03-29T10:51:49.733 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.744+0000 7f28ae1f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:49.733 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.744+0000 7f28ae1f6640 1 -- 172.21.15.112:0/4015377083 >> v1:172.21.15.118:6789/0 conn(0x7f28a813f650 legacy=0x7f28a80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:49.733 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.747+0000 7f28ae1f6640 1 -- 172.21.15.112:0/4015377083 shutdown_connections 2024-03-29T10:51:49.733 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.747+0000 7f28ae1f6640 1 -- 172.21.15.112:0/4015377083 >> 172.21.15.112:0/4015377083 conn(0x7f28a8066a00 msgr2=0x7f28a8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:49.733 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.747+0000 7f28ae1f6640 1 -- 172.21.15.112:0/4015377083 shutdown_connections 2024-03-29T10:51:49.733 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.747+0000 7f28ae1f6640 1 -- 172.21.15.112:0/4015377083 wait complete. 2024-03-29T10:51:49.736 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.748+0000 7f28ae1f6640 10 monclient: build_initial_monmap 2024-03-29T10:51:49.736 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.748+0000 7f28ae1f6640 10 monclient: monmap: 2024-03-29T10:51:49.736 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:51:49.736 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:49.736 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:51:49.736 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:51:49.736 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:49.736 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:49.736 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:49.736 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:51:49.737 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:51:49.737 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:49.737 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.749+0000 7f28ae1f6640 1 librados: starting msgr at 2024-03-29T10:51:49.737 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.749+0000 7f28ae1f6640 1 librados: starting objecter 2024-03-29T10:51:49.737 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.749+0000 7f28ae1f6640 1 Processor -- start 2024-03-29T10:51:49.737 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.749+0000 7f28ae1f6640 1 -- start start 2024-03-29T10:51:49.737 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.749+0000 7f28ae1f6640 1 librados: setting wanted keys 2024-03-29T10:51:49.737 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.749+0000 7f28ae1f6640 1 librados: calling monclient init 2024-03-29T10:51:49.737 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.749+0000 7f28ae1f6640 10 monclient: init 2024-03-29T10:51:49.737 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.750+0000 7f28ae1f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:49.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.750+0000 7f28ae1f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:49.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.752+0000 7f28ae1f6640 10 monclient(hunting): picked mon.noname-a con 0x7f28a80a9d70 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:49.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.752+0000 7f28ae1f6640 10 monclient(hunting): picked mon.noname-b con 0x7f28a813f650 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:49.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.752+0000 7f28ae1f6640 10 monclient(hunting): picked mon.noname-c con 0x7f28a81457f0 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:49.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.752+0000 7f28ae1f6640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a80abc30 con 0x7f28a80a9d70 2024-03-29T10:51:49.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.752+0000 7f28ae1f6640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a80abea0 con 0x7f28a813f650 2024-03-29T10:51:49.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.752+0000 7f28ae1f6640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a80ac180 con 0x7f28a81457f0 2024-03-29T10:51:49.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.752+0000 7f28ae1f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:49.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.752+0000 7f28ae1f6640 10 monclient(hunting): authenticate will time out at 865.056910s 2024-03-29T10:51:49.739 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.756+0000 7f28a77fe640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f28a80a9d70 0x7f28a8071190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:52245/0 (socket says 172.21.15.112:52245) 2024-03-29T10:51:49.739 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.756+0000 7f28a77fe640 1 -- 172.21.15.112:0/575421929 learned_addr learned my addr 172.21.15.112:0/575421929 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:51:49.741 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.760+0000 7f288ffff640 10 client.?.objecter ms_handle_connect 0x7f28a80a9d70 2024-03-29T10:51:49.741 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.760+0000 7f288ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:49.741 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.760+0000 7f288ffff640 1 -- 172.21.15.112:0/575421929 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2838394 0 0) 0x7f28a80abc30 con 0x7f28a80a9d70 2024-03-29T10:51:49.741 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.760+0000 7f288ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:49.741 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.760+0000 7f288ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:49.742 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.760+0000 7f288ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:49.742 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.760+0000 7f288ffff640 10 monclient(hunting): my global_id is 4245 2024-03-29T10:51:49.742 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.760+0000 7f288ffff640 1 -- 172.21.15.112:0/575421929 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2894004eb0 con 0x7f28a80a9d70 2024-03-29T10:51:49.742 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.760+0000 7f288ffff640 1 -- 172.21.15.112:0/575421929 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 832171334 0 0) 0x7f2894004eb0 con 0x7f28a80a9d70 2024-03-29T10:51:49.742 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.760+0000 7f288ffff640 1 -- 172.21.15.112:0/575421929 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28a80abc30 con 0x7f28a80a9d70 2024-03-29T10:51:49.742 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.760+0000 7f288ffff640 1 -- 172.21.15.112:0/575421929 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f289c0017d0 con 0x7f28a80a9d70 2024-03-29T10:51:49.742 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f288ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:49.742 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f288ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:49.742 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f288ffff640 10 monclient(hunting): dump: 2024-03-29T10:51:49.743 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:49.743 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:49.743 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:49.743 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:49.743 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:49.743 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:49.743 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:49.743 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:49.743 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:49.743 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:49.743 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f288ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:49.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f288ffff640 10 client.?.objecter ms_handle_connect 0x7f28a813f650 2024-03-29T10:51:49.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f288ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:49.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f288ffff640 1 -- 172.21.15.112:0/575421929 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3605584513 0 0) 0x7f28a80abc30 con 0x7f28a80a9d70 2024-03-29T10:51:49.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f288ffff640 1 -- 172.21.15.112:0/575421929 >> v1:172.21.15.112:6790/0 conn(0x7f28a81457f0 legacy=0x7f28a81e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:49.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f288ffff640 1 -- 172.21.15.112:0/575421929 >> v1:172.21.15.118:6789/0 conn(0x7f28a813f650 legacy=0x7f28a80706c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:49.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f288ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:51:49.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f288ffff640 1 monclient: found mon.a 2024-03-29T10:51:49.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f288ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f288ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f288ffff640 1 -- 172.21.15.112:0/575421929 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a8133520 con 0x7f28a80a9d70 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f288ffff640 10 monclient: _finish_auth 0 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f288ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f288ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f28ae1f6640 5 monclient: authenticate success, global_id 4245 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f28ae1f6640 10 monclient: _renew_subs 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f28ae1f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f28ae1f6640 1 -- 172.21.15.112:0/575421929 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a81327d0 con 0x7f28a80a9d70 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f28ae1f6640 10 client.4245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f28ae1f6640 10 monclient: _renew_subs 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f28ae1f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f28ae1f6640 1 -- 172.21.15.112:0/575421929 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a8070dd0 con 0x7f28a80a9d70 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.761+0000 7f28ae1f6640 1 librados: init done 2024-03-29T10:51:49.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.762+0000 7f288ffff640 1 -- 172.21.15.112:0/575421929 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f289c001930 con 0x7f28a80a9d70 2024-03-29T10:51:49.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.762+0000 7f288ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:49.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.762+0000 7f288ffff640 1 -- 172.21.15.112:0/575421929 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f289c004a40 con 0x7f28a80a9d70 2024-03-29T10:51:49.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.762+0000 7f288ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:49.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.762+0000 7f288ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:49.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.762+0000 7f288ffff640 10 monclient: dump: 2024-03-29T10:51:49.746 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:49.746 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:49.746 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:49.746 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:49.746 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:49.746 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:49.746 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:49.746 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:49.747 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:49.747 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:49.748 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.763+0000 7f28a5ffb640 4 set_mon_vals no callback set 2024-03-29T10:51:49.748 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.765+0000 7f288ffff640 1 -- 172.21.15.112:0/575421929 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f289c004ba0 con 0x7f28a80a9d70 2024-03-29T10:51:49.748 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.766+0000 7f28ae1f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:49.748 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.766+0000 7f28ae1f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:49.748 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.766+0000 7f288ffff640 1 -- 172.21.15.112:0/575421929 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f289c084ac0 con 0x7f28a80a9d70 2024-03-29T10:51:49.748 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.766+0000 7f288ffff640 10 client.4245.objecter ms_dispatch 0x7f28a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:49.748 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.766+0000 7f288ffff640 3 client.4245.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:49.748 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.766+0000 7f288ffff640 3 client.4245.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:49.749 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.766+0000 7f288ffff640 20 client.4245.objecter dump_active .. 0 homeless 2024-03-29T10:51:49.749 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.766+0000 7f28ae1f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:51:49.749 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.766+0000 7f28ae1f6640 1 -- 172.21.15.112:0/575421929 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2888005080 con 0x7f28a80a9d70 2024-03-29T10:51:49.751 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.769+0000 7f28a6ffd640 10 monclient: get_auth_request con 0x7f289406dca0 auth_method 0 2024-03-29T10:51:49.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.772+0000 7f288ffff640 10 client.4245.objecter ms_handle_connect 0x7f289406dca0 2024-03-29T10:51:49.758 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.776+0000 7f288ffff640 1 -- 172.21.15.112:0/575421929 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f289c0501d0 con 0x7f28a80a9d70 2024-03-29T10:51:49.758 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.776+0000 7f288ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:49.758 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:49.776+0000 7f288ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:50.086 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:50.104+0000 7f28ae1f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:50.336 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:50.104+0000 7f28ae1f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:50.336 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:50.104+0000 7f28ae1f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:51:50.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:50.104+0000 7f28ae1f6640 1 -- 172.21.15.112:0/575421929 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2888005330 con 0x7f28a80a9d70 2024-03-29T10:51:50.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:50.106+0000 7f288ffff640 1 -- 172.21.15.112:0/575421929 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f289c0522d0 con 0x7f28a80a9d70 2024-03-29T10:51:50.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:50.106+0000 7f288ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:50.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:50.106+0000 7f288ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:50.338 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:51:50.338 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:50.339 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:50.110+0000 7f28ae1f6640 10 librados: watch_flush enter 2024-03-29T10:51:50.339 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:50.110+0000 7f28ae1f6640 10 librados: watch_flush exit 2024-03-29T10:51:50.339 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:50.110+0000 7f28ae1f6640 20 client.4245.objecter shutdown clearing up homeless session... 2024-03-29T10:51:50.339 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:50.110+0000 7f28ae1f6640 10 client.4245.objecter successfully canceled tick 2024-03-29T10:51:50.339 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:50.110+0000 7f28ae1f6640 1 -- 172.21.15.112:0/575421929 >> v1:172.21.15.118:6816/3181073408 conn(0x7f289406dca0 legacy=0x7f2894070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:50.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:50.110+0000 7f28ae1f6640 10 monclient: shutdown 2024-03-29T10:51:50.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:50.110+0000 7f28ae1f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:50.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:50.110+0000 7f28ae1f6640 1 -- 172.21.15.112:0/575421929 >> v1:172.21.15.112:6789/0 conn(0x7f28a80a9d70 legacy=0x7f28a8071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:50.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:50.112+0000 7f28ae1f6640 1 -- 172.21.15.112:0/575421929 shutdown_connections 2024-03-29T10:51:50.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:50.112+0000 7f28ae1f6640 1 -- 172.21.15.112:0/575421929 >> 172.21.15.112:0/575421929 conn(0x7f28a8066a00 msgr2=0x7f28a80b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:50.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:50.114+0000 7f28ae1f6640 1 -- 172.21.15.112:0/575421929 shutdown_connections 2024-03-29T10:51:50.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:50.114+0000 7f28ae1f6640 1 -- 172.21.15.112:0/575421929 wait complete. 2024-03-29T10:51:50.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:50.116+0000 7f28ae1f6640 1 librados: shutdown 2024-03-29T10:51:50.340 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:57.148 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:57.303 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.321+0000 7faa71277640 10 monclient: get_monmap_and_config 2024-03-29T10:51:57.304 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.321+0000 7faa71277640 10 monclient: build_initial_monmap 2024-03-29T10:51:57.304 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.321+0000 7faa71277640 10 monclient: monmap: 2024-03-29T10:51:57.304 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:51:57.304 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:57.304 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:51:57.322252+0000 2024-03-29T10:51:57.304 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:51:57.322252+0000 2024-03-29T10:51:57.304 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:57.304 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:57.304 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:57.304 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:51:57.304 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:51:57.304 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:57.305 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.322+0000 7faa71277640 1 Processor -- start 2024-03-29T10:51:57.305 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.323+0000 7faa71277640 1 -- start start 2024-03-29T10:51:57.305 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.323+0000 7faa71277640 10 monclient: init 2024-03-29T10:51:57.306 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.324+0000 7faa71277640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:57.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.324+0000 7faa71277640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:57.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.324+0000 7faa71277640 10 monclient(hunting): picked mon.noname-a con 0x7faa64006410 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:57.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.324+0000 7faa71277640 10 monclient(hunting): picked mon.noname-c con 0x7faa6402a640 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:57.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.324+0000 7faa71277640 10 monclient(hunting): picked mon.noname-b con 0x7faa6402b370 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:57.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.324+0000 7faa71277640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa6402a120 con 0x7faa64006410 2024-03-29T10:51:57.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.324+0000 7faa71277640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa640c75b0 con 0x7faa6402a640 2024-03-29T10:51:57.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.324+0000 7faa71277640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa64033110 con 0x7faa6402b370 2024-03-29T10:51:57.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.324+0000 7faa71277640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:57.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.324+0000 7faa71277640 10 monclient(hunting): authenticate will time out at 872.629110s 2024-03-29T10:51:57.313 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.331+0000 7faa6bfff640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7faa64006410 0x7faa640345e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:45757/0 (socket says 172.21.15.112:45757) 2024-03-29T10:51:57.313 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.331+0000 7faa6bfff640 1 -- 172.21.15.112:0/3686326919 learned_addr learned my addr 172.21.15.112:0/3686326919 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:51:57.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.334+0000 7faa6affd640 1 -- 172.21.15.112:0/3686326919 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116276201 0 0) 0x7faa6402a120 con 0x7faa64006410 2024-03-29T10:51:57.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:57.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:57.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:57.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 10 monclient(hunting): my global_id is 4248 2024-03-29T10:51:57.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 1 -- 172.21.15.112:0/3686326919 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa5c003520 con 0x7faa64006410 2024-03-29T10:51:57.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 1 -- 172.21.15.112:0/3686326919 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1941662194 0 0) 0x7faa5c003520 con 0x7faa64006410 2024-03-29T10:51:57.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 1 -- 172.21.15.112:0/3686326919 >> v1:172.21.15.112:6790/0 conn(0x7faa6402b370 legacy=0x7faa640320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:57.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 1 -- 172.21.15.112:0/3686326919 >> v1:172.21.15.118:6789/0 conn(0x7faa6402a640 legacy=0x7faa640c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:57.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:57.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 1 monclient: found mon.noname-a 2024-03-29T10:51:57.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:57.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T10:51:57.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 1 -- 172.21.15.112:0/3686326919 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa640332a0 con 0x7faa64006410 2024-03-29T10:51:57.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 10 monclient: _finish_auth 0 2024-03-29T10:51:57.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:57.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:57.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 1 -- 172.21.15.112:0/3686326919 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7faa60001690 con 0x7faa64006410 2024-03-29T10:51:57.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:57.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:57.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 10 monclient: dump: 2024-03-29T10:51:57.318 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:57.318 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:57.318 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:57.318 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:57.319 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:57.319 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:57.319 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:57.319 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:57.319 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:57.319 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:57.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa71277640 5 monclient: authenticate success, global_id 4248 2024-03-29T10:51:57.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa71277640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:57.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 1 -- 172.21.15.112:0/3686326919 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa600017f0 con 0x7faa64006410 2024-03-29T10:51:57.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:57.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 1 -- 172.21.15.112:0/3686326919 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7faa60001bc0 con 0x7faa64006410 2024-03-29T10:51:57.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:57.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:57.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.335+0000 7faa6affd640 10 monclient: dump: 2024-03-29T10:51:57.320 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:57.320 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:57.320 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:57.320 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:57.320 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:57.320 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:57.320 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:57.320 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:57.320 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:57.320 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:57.322 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.338+0000 7faa71277640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:57.322 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.338+0000 7faa71277640 4 set_mon_vals no callback set 2024-03-29T10:51:57.322 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.339+0000 7faa71277640 10 monclient: shutdown 2024-03-29T10:51:57.322 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.339+0000 7faa71277640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:57.323 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.339+0000 7faa71277640 1 -- 172.21.15.112:0/3686326919 >> v1:172.21.15.112:6789/0 conn(0x7faa64006410 legacy=0x7faa640345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:57.323 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.339+0000 7faa71277640 1 -- 172.21.15.112:0/3686326919 shutdown_connections 2024-03-29T10:51:57.323 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.339+0000 7faa71277640 1 -- 172.21.15.112:0/3686326919 >> 172.21.15.112:0/3686326919 conn(0x7faa640058f0 msgr2=0x7faa64005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:57.323 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.340+0000 7faa71277640 1 -- 172.21.15.112:0/3686326919 shutdown_connections 2024-03-29T10:51:57.323 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.340+0000 7faa71277640 1 -- 172.21.15.112:0/3686326919 wait complete. 2024-03-29T10:51:57.324 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.340+0000 7faa71277640 10 monclient: build_initial_monmap 2024-03-29T10:51:57.324 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.340+0000 7faa71277640 10 monclient: monmap: 2024-03-29T10:51:57.324 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:51:57.324 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:57.324 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:51:57.324 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:51:57.324 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:57.324 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:57.324 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:51:57.324 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:51:57.324 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:51:57.324 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:57.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.341+0000 7faa71277640 1 librados: starting msgr at 2024-03-29T10:51:57.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.341+0000 7faa71277640 1 librados: starting objecter 2024-03-29T10:51:57.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.341+0000 7faa71277640 1 Processor -- start 2024-03-29T10:51:57.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.342+0000 7faa71277640 1 -- start start 2024-03-29T10:51:57.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.342+0000 7faa71277640 1 librados: setting wanted keys 2024-03-29T10:51:57.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.342+0000 7faa71277640 1 librados: calling monclient init 2024-03-29T10:51:57.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.342+0000 7faa71277640 10 monclient: init 2024-03-29T10:51:57.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.342+0000 7faa71277640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:57.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.342+0000 7faa71277640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:57.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.343+0000 7faa71277640 10 monclient(hunting): picked mon.noname-c con 0x7faa64006410 addr v1:172.21.15.112:6790/0 2024-03-29T10:51:57.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.343+0000 7faa71277640 10 monclient(hunting): picked mon.noname-a con 0x7faa6402a640 addr v1:172.21.15.112:6789/0 2024-03-29T10:51:57.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.343+0000 7faa71277640 10 monclient(hunting): picked mon.noname-b con 0x7faa6402b370 addr v1:172.21.15.118:6789/0 2024-03-29T10:51:57.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.343+0000 7faa71277640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa640b27c0 con 0x7faa6402a640 2024-03-29T10:51:57.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.343+0000 7faa71277640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa6402cde0 con 0x7faa6402b370 2024-03-29T10:51:57.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.343+0000 7faa71277640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa6402d0c0 con 0x7faa64006410 2024-03-29T10:51:57.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.343+0000 7faa71277640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:57.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.343+0000 7faa71277640 10 monclient(hunting): authenticate will time out at 872.647546s 2024-03-29T10:51:57.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.346+0000 7faa6b7fe640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7faa6402a640 0x7faa640b19a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:45863/0 (socket says 172.21.15.112:45863) 2024-03-29T10:51:57.329 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.346+0000 7faa6b7fe640 1 -- 172.21.15.112:0/649348766 learned_addr learned my addr 172.21.15.112:0/649348766 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:51:57.329 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.347+0000 7faa68ff9640 10 client.?.objecter ms_handle_connect 0x7faa64006410 2024-03-29T10:51:57.329 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.347+0000 7faa68ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:57.329 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.347+0000 7faa68ff9640 1 -- 172.21.15.112:0/649348766 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813171188 0 0) 0x7faa6402d0c0 con 0x7faa64006410 2024-03-29T10:51:57.329 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.347+0000 7faa68ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:57.329 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.347+0000 7faa68ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:57.329 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.347+0000 7faa68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:57.329 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.347+0000 7faa68ff9640 10 monclient(hunting): my global_id is 4253 2024-03-29T10:51:57.329 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.347+0000 7faa68ff9640 1 -- 172.21.15.112:0/649348766 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa50004eb0 con 0x7faa64006410 2024-03-29T10:51:57.330 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.347+0000 7faa68ff9640 1 -- 172.21.15.112:0/649348766 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1481497879 0 0) 0x7faa50004eb0 con 0x7faa64006410 2024-03-29T10:51:57.330 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.347+0000 7faa68ff9640 1 -- 172.21.15.112:0/649348766 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa6402d0c0 con 0x7faa64006410 2024-03-29T10:51:57.330 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.347+0000 7faa68ff9640 1 -- 172.21.15.112:0/649348766 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7faa60001bc0 con 0x7faa64006410 2024-03-29T10:51:57.330 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.347+0000 7faa68ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:57.330 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.347+0000 7faa68ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:57.330 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.347+0000 7faa68ff9640 10 monclient(hunting): dump: 2024-03-29T10:51:57.330 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:57.330 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:57.330 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:57.331 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:57.331 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:57.331 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:57.331 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:57.331 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:57.331 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:57.331 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:57.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.347+0000 7faa68ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:57.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.347+0000 7faa68ff9640 1 -- 172.21.15.112:0/649348766 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3309859160 0 0) 0x7faa6402d0c0 con 0x7faa64006410 2024-03-29T10:51:57.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.347+0000 7faa68ff9640 1 -- 172.21.15.112:0/649348766 >> v1:172.21.15.118:6789/0 conn(0x7faa6402b370 legacy=0x7faa640b20b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:51:57.336 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.351+0000 7faa68ff9640 1 -- 172.21.15.112:0/649348766 >> v1:172.21.15.112:6789/0 conn(0x7faa6402a640 legacy=0x7faa640b19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:57.336 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.351+0000 7faa68ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:51:57.336 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.351+0000 7faa68ff9640 1 monclient: found mon.c 2024-03-29T10:51:57.336 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.351+0000 7faa68ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:57.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.351+0000 7faa68ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:51:57.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.351+0000 7faa68ff9640 1 -- 172.21.15.112:0/649348766 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa6402d3a0 con 0x7faa64006410 2024-03-29T10:51:57.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.351+0000 7faa68ff9640 10 monclient: _finish_auth 0 2024-03-29T10:51:57.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.351+0000 7faa68ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:51:57.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.351+0000 7faa68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:57.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.351+0000 7faa68ff9640 10 client.?.objecter ms_handle_connect 0x7faa6402a640 2024-03-29T10:51:57.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.351+0000 7faa68ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:57.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.351+0000 7faa71277640 5 monclient: authenticate success, global_id 4253 2024-03-29T10:51:57.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.351+0000 7faa71277640 10 monclient: _renew_subs 2024-03-29T10:51:57.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.351+0000 7faa71277640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:51:57.338 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.351+0000 7faa71277640 1 -- 172.21.15.112:0/649348766 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa640c38c0 con 0x7faa64006410 2024-03-29T10:51:57.338 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.351+0000 7faa71277640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:57.338 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.351+0000 7faa71277640 10 monclient: _renew_subs 2024-03-29T10:51:57.338 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.351+0000 7faa71277640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:51:57.338 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.351+0000 7faa71277640 1 -- 172.21.15.112:0/649348766 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa640bd680 con 0x7faa64006410 2024-03-29T10:51:57.338 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.352+0000 7faa71277640 1 librados: init done 2024-03-29T10:51:57.338 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.352+0000 7faa68ff9640 1 -- 172.21.15.112:0/649348766 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa60004a90 con 0x7faa64006410 2024-03-29T10:51:57.339 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.352+0000 7faa68ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:57.339 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.352+0000 7faa68ff9640 1 -- 172.21.15.112:0/649348766 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7faa60004e60 con 0x7faa64006410 2024-03-29T10:51:57.339 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.352+0000 7faa68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:57.339 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.352+0000 7faa68ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:57.339 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.352+0000 7faa68ff9640 10 monclient: dump: 2024-03-29T10:51:57.339 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:51:57.340 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:51:57.340 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:57.340 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:51:57.340 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:57.340 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:51:57.340 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:51:57.341 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:51:57.341 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:51:57.341 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:51:57.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.355+0000 7faa6a7fc640 4 set_mon_vals no callback set 2024-03-29T10:51:57.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.355+0000 7faa68ff9640 1 -- 172.21.15.112:0/649348766 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7faa60004fc0 con 0x7faa64006410 2024-03-29T10:51:57.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.356+0000 7faa68ff9640 1 -- 172.21.15.112:0/649348766 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7faa60084c60 con 0x7faa64006410 2024-03-29T10:51:57.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.356+0000 7faa68ff9640 10 client.4253.objecter ms_dispatch 0x7faa6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:57.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.356+0000 7faa68ff9640 3 client.4253.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:57.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.356+0000 7faa68ff9640 3 client.4253.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:57.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.356+0000 7faa68ff9640 20 client.4253.objecter dump_active .. 0 homeless 2024-03-29T10:51:57.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.356+0000 7faa71277640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:57.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.356+0000 7faa71277640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:57.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.356+0000 7faa71277640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:51:57.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.356+0000 7faa71277640 1 -- 172.21.15.112:0/649348766 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa4c005080 con 0x7faa64006410 2024-03-29T10:51:57.344 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.356+0000 7faa6b7fe640 10 monclient: get_auth_request con 0x7faa50076390 auth_method 0 2024-03-29T10:51:57.344 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.359+0000 7faa68ff9640 10 client.4253.objecter ms_handle_connect 0x7faa50076390 2024-03-29T10:51:57.348 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.366+0000 7faa68ff9640 1 -- 172.21.15.112:0/649348766 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faa60050540 con 0x7faa64006410 2024-03-29T10:51:57.348 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.366+0000 7faa68ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:57.348 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.366+0000 7faa68ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:57.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.697+0000 7faa71277640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:57.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.698+0000 7faa71277640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:57.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.698+0000 7faa71277640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:51:57.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.698+0000 7faa71277640 1 -- 172.21.15.112:0/649348766 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa4c005330 con 0x7faa64006410 2024-03-29T10:51:57.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.698+0000 7faa68ff9640 1 -- 172.21.15.112:0/649348766 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7faa60052470 con 0x7faa64006410 2024-03-29T10:51:57.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.698+0000 7faa68ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:57.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.698+0000 7faa68ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:57.681 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:51:57.681 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:57.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.703+0000 7faa71277640 10 librados: watch_flush enter 2024-03-29T10:51:57.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.704+0000 7faa71277640 10 librados: watch_flush exit 2024-03-29T10:51:57.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.704+0000 7faa71277640 20 client.4253.objecter shutdown clearing up homeless session... 2024-03-29T10:51:57.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.704+0000 7faa71277640 10 client.4253.objecter successfully canceled tick 2024-03-29T10:51:57.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.704+0000 7faa71277640 1 -- 172.21.15.112:0/649348766 >> v1:172.21.15.118:6816/3181073408 conn(0x7faa50076390 legacy=0x7faa50078850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:57.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.704+0000 7faa71277640 10 monclient: shutdown 2024-03-29T10:51:57.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.704+0000 7faa71277640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:57.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.704+0000 7faa71277640 1 -- 172.21.15.112:0/649348766 >> v1:172.21.15.112:6790/0 conn(0x7faa64006410 legacy=0x7faa6402d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:57.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.704+0000 7faa71277640 1 -- 172.21.15.112:0/649348766 shutdown_connections 2024-03-29T10:51:57.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.704+0000 7faa71277640 1 -- 172.21.15.112:0/649348766 >> 172.21.15.112:0/649348766 conn(0x7faa640058f0 msgr2=0x7faa640043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:57.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.705+0000 7faa71277640 1 -- 172.21.15.112:0/649348766 shutdown_connections 2024-03-29T10:51:57.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.705+0000 7faa71277640 1 -- 172.21.15.112:0/649348766 wait complete. 2024-03-29T10:51:57.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:51:57.708+0000 7faa71277640 1 librados: shutdown 2024-03-29T10:51:57.710 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:04.517 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:04.667 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.685+0000 7f9c154ff640 10 monclient: get_monmap_and_config 2024-03-29T10:52:04.667 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.685+0000 7f9c154ff640 10 monclient: build_initial_monmap 2024-03-29T10:52:04.667 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.685+0000 7f9c154ff640 10 monclient: monmap: 2024-03-29T10:52:04.667 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:52:04.667 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:04.667 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:52:04.686343+0000 2024-03-29T10:52:04.667 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:52:04.686343+0000 2024-03-29T10:52:04.667 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:04.667 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:04.667 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:52:04.667 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:52:04.668 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:52:04.668 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:04.668 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.686+0000 7f9c154ff640 1 Processor -- start 2024-03-29T10:52:04.669 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.686+0000 7f9c154ff640 1 -- start start 2024-03-29T10:52:04.669 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.686+0000 7f9c154ff640 10 monclient: init 2024-03-29T10:52:04.669 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.687+0000 7f9c154ff640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:04.669 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.687+0000 7f9c154ff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:04.669 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.687+0000 7f9c154ff640 10 monclient(hunting): picked mon.noname-b con 0x7f9c101422b0 addr v1:172.21.15.112:6790/0 2024-03-29T10:52:04.669 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.687+0000 7f9c154ff640 10 monclient(hunting): picked mon.noname-c con 0x7f9c10142690 addr v1:172.21.15.118:6789/0 2024-03-29T10:52:04.669 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.687+0000 7f9c154ff640 10 monclient(hunting): picked mon.noname-a con 0x7f9c100b17d0 addr v1:172.21.15.112:6789/0 2024-03-29T10:52:04.669 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.687+0000 7f9c154ff640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c10142c80 con 0x7f9c100b17d0 2024-03-29T10:52:04.669 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.687+0000 7f9c154ff640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c100aae40 con 0x7f9c10142690 2024-03-29T10:52:04.669 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.687+0000 7f9c154ff640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1013fbd0 con 0x7f9c101422b0 2024-03-29T10:52:04.669 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.687+0000 7f9c154ff640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:04.669 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.687+0000 7f9c154ff640 10 monclient(hunting): authenticate will time out at 879.991582s 2024-03-29T10:52:04.671 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.690+0000 7f9c0f7fe640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f9c100b17d0 0x7f9c100b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35135/0 (socket says 172.21.15.112:35135) 2024-03-29T10:52:04.671 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.690+0000 7f9c0f7fe640 1 -- 172.21.15.112:0/640013172 learned_addr learned my addr 172.21.15.112:0/640013172 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:52:04.673 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.691+0000 7f9c0dffb640 1 -- 172.21.15.112:0/640013172 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1421417859 0 0) 0x7f9c10142c80 con 0x7f9c100b17d0 2024-03-29T10:52:04.673 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.691+0000 7f9c0dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:04.673 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.691+0000 7f9c0dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:04.673 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.691+0000 7f9c0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:04.673 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.691+0000 7f9c0dffb640 10 monclient(hunting): my global_id is 4254 2024-03-29T10:52:04.673 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.691+0000 7f9c0dffb640 1 -- 172.21.15.112:0/640013172 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf8003520 con 0x7f9c100b17d0 2024-03-29T10:52:04.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.693+0000 7f9c0dffb640 1 -- 172.21.15.112:0/640013172 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3935726572 0 0) 0x7f9bf8003520 con 0x7f9c100b17d0 2024-03-29T10:52:04.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.693+0000 7f9c0dffb640 1 -- 172.21.15.112:0/640013172 >> v1:172.21.15.112:6790/0 conn(0x7f9c101422b0 legacy=0x7f9c1013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:04.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.693+0000 7f9c0dffb640 1 -- 172.21.15.112:0/640013172 >> v1:172.21.15.118:6789/0 conn(0x7f9c10142690 legacy=0x7f9c100b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:52:04.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.693+0000 7f9c0dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:52:04.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.693+0000 7f9c0dffb640 1 monclient: found mon.noname-a 2024-03-29T10:52:04.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.693+0000 7f9c0dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:04.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.693+0000 7f9c0dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T10:52:04.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.693+0000 7f9c0dffb640 1 -- 172.21.15.112:0/640013172 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c10151940 con 0x7f9c100b17d0 2024-03-29T10:52:04.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.693+0000 7f9c0dffb640 10 monclient: _finish_auth 0 2024-03-29T10:52:04.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.693+0000 7f9c0dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:52:04.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.693+0000 7f9c0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:04.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.693+0000 7f9c0dffb640 1 -- 172.21.15.112:0/640013172 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9c08001a10 con 0x7f9c100b17d0 2024-03-29T10:52:04.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.693+0000 7f9c0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:04.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.693+0000 7f9c0dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:04.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.693+0000 7f9c0dffb640 10 monclient: dump: 2024-03-29T10:52:04.681 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:04.681 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:04.681 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:04.681 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:04.681 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:04.681 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:04.681 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:04.681 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:04.681 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:04.681 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:04.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.694+0000 7f9c0dffb640 1 -- 172.21.15.112:0/640013172 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c08001b70 con 0x7f9c100b17d0 2024-03-29T10:52:04.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.694+0000 7f9c0dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:04.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.694+0000 7f9c0dffb640 1 -- 172.21.15.112:0/640013172 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9c08001f80 con 0x7f9c100b17d0 2024-03-29T10:52:04.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.694+0000 7f9c0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:04.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.694+0000 7f9c0dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:04.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.694+0000 7f9c0dffb640 10 monclient: dump: 2024-03-29T10:52:04.682 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:04.682 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:04.682 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:04.682 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:04.682 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:04.682 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:04.682 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:04.682 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:04.683 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:04.683 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:04.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.695+0000 7f9c154ff640 5 monclient: authenticate success, global_id 4254 2024-03-29T10:52:04.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.695+0000 7f9c154ff640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:04.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.695+0000 7f9c154ff640 4 set_mon_vals no callback set 2024-03-29T10:52:04.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.696+0000 7f9c154ff640 10 monclient: shutdown 2024-03-29T10:52:04.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.696+0000 7f9c154ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:04.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.696+0000 7f9c154ff640 1 -- 172.21.15.112:0/640013172 >> v1:172.21.15.112:6789/0 conn(0x7f9c100b17d0 legacy=0x7f9c100b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:04.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.696+0000 7f9c154ff640 1 -- 172.21.15.112:0/640013172 shutdown_connections 2024-03-29T10:52:04.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.696+0000 7f9c154ff640 1 -- 172.21.15.112:0/640013172 >> 172.21.15.112:0/640013172 conn(0x7f9c10066a00 msgr2=0x7f9c10066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:04.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.698+0000 7f9c154ff640 1 -- 172.21.15.112:0/640013172 shutdown_connections 2024-03-29T10:52:04.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.699+0000 7f9c154ff640 1 -- 172.21.15.112:0/640013172 wait complete. 2024-03-29T10:52:04.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.700+0000 7f9c154ff640 10 monclient: build_initial_monmap 2024-03-29T10:52:04.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.700+0000 7f9c154ff640 10 monclient: monmap: 2024-03-29T10:52:04.684 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:52:04.684 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:04.684 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:52:04.684 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:52:04.684 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:04.684 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:04.684 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:52:04.684 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:52:04.684 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:52:04.685 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:04.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.703+0000 7f9c154ff640 1 librados: starting msgr at 2024-03-29T10:52:04.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.703+0000 7f9c154ff640 1 librados: starting objecter 2024-03-29T10:52:04.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.703+0000 7f9c154ff640 1 Processor -- start 2024-03-29T10:52:04.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.704+0000 7f9c154ff640 1 -- start start 2024-03-29T10:52:04.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.704+0000 7f9c154ff640 1 librados: setting wanted keys 2024-03-29T10:52:04.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.704+0000 7f9c154ff640 1 librados: calling monclient init 2024-03-29T10:52:04.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.704+0000 7f9c154ff640 10 monclient: init 2024-03-29T10:52:04.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.704+0000 7f9c154ff640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:04.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.705+0000 7f9c154ff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:04.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.705+0000 7f9c154ff640 10 monclient(hunting): picked mon.noname-b con 0x7f9c100b17d0 addr v1:172.21.15.118:6789/0 2024-03-29T10:52:04.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.705+0000 7f9c154ff640 10 monclient(hunting): picked mon.noname-a con 0x7f9c101422b0 addr v1:172.21.15.112:6789/0 2024-03-29T10:52:04.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.705+0000 7f9c154ff640 10 monclient(hunting): picked mon.noname-c con 0x7f9c10142690 addr v1:172.21.15.112:6790/0 2024-03-29T10:52:04.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.708+0000 7f9c154ff640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1013c250 con 0x7f9c101422b0 2024-03-29T10:52:04.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.708+0000 7f9c154ff640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1013c500 con 0x7f9c100b17d0 2024-03-29T10:52:04.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.708+0000 7f9c154ff640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1013c7e0 con 0x7f9c10142690 2024-03-29T10:52:04.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.708+0000 7f9c154ff640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:04.690 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.708+0000 7f9c154ff640 10 monclient(hunting): authenticate will time out at 880.012568s 2024-03-29T10:52:04.690 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.709+0000 7f9c0f7fe640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f9c10142690 0x7f9c101e6e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35517/0 (socket says 172.21.15.112:35517) 2024-03-29T10:52:04.690 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.709+0000 7f9c0f7fe640 1 -- 172.21.15.112:0/2657891813 learned_addr learned my addr 172.21.15.112:0/2657891813 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:52:04.691 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.709+0000 7f9bf77fe640 10 client.?.objecter ms_handle_connect 0x7f9c101422b0 2024-03-29T10:52:04.691 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.709+0000 7f9bf77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:04.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.709+0000 7f9bf77fe640 10 client.?.objecter ms_handle_connect 0x7f9c10142690 2024-03-29T10:52:04.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.709+0000 7f9bf77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:04.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.709+0000 7f9bf77fe640 1 -- 172.21.15.112:0/2657891813 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442142167 0 0) 0x7f9c1013c250 con 0x7f9c101422b0 2024-03-29T10:52:04.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.709+0000 7f9bf77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:04.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.709+0000 7f9bf77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:04.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.709+0000 7f9bf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:04.698 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.709+0000 7f9bf77fe640 10 monclient(hunting): my global_id is 4257 2024-03-29T10:52:04.698 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.709+0000 7f9bf77fe640 1 -- 172.21.15.112:0/2657891813 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bfc004eb0 con 0x7f9c101422b0 2024-03-29T10:52:04.698 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.709+0000 7f9bf77fe640 1 -- 172.21.15.112:0/2657891813 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849923071 0 0) 0x7f9c1013c7e0 con 0x7f9c10142690 2024-03-29T10:52:04.698 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.709+0000 7f9bf77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:04.698 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.709+0000 7f9bf77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:04.698 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.709+0000 7f9bf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:04.699 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.709+0000 7f9bf77fe640 10 monclient(hunting): my global_id is 4256 2024-03-29T10:52:04.699 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.710+0000 7f9bf77fe640 1 -- 172.21.15.112:0/2657891813 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c1013c250 con 0x7f9c10142690 2024-03-29T10:52:04.699 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.710+0000 7f9bf77fe640 1 -- 172.21.15.112:0/2657891813 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 853297250 0 0) 0x7f9bfc004eb0 con 0x7f9c101422b0 2024-03-29T10:52:04.699 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.710+0000 7f9bf77fe640 1 -- 172.21.15.112:0/2657891813 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c1013c7e0 con 0x7f9c101422b0 2024-03-29T10:52:04.699 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.710+0000 7f9bf77fe640 1 -- 172.21.15.112:0/2657891813 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9c000017d0 con 0x7f9c101422b0 2024-03-29T10:52:04.699 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.710+0000 7f9bf77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:04.699 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.710+0000 7f9bf77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:04.699 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.710+0000 7f9bf77fe640 10 monclient(hunting): dump: 2024-03-29T10:52:04.699 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:04.699 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:04.700 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:04.700 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:04.700 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:04.700 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:04.700 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:04.700 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:04.700 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:04.700 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:04.700 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.710+0000 7f9bf77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:04.700 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.711+0000 7f9bf77fe640 1 -- 172.21.15.112:0/2657891813 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3686573530 0 0) 0x7f9c1013c250 con 0x7f9c10142690 2024-03-29T10:52:04.700 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.711+0000 7f9bf77fe640 1 -- 172.21.15.112:0/2657891813 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bfc004eb0 con 0x7f9c10142690 2024-03-29T10:52:04.700 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.711+0000 7f9bf77fe640 1 -- 172.21.15.112:0/2657891813 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9c08001e80 con 0x7f9c10142690 2024-03-29T10:52:04.700 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.711+0000 7f9bf77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:04.701 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.711+0000 7f9bf77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:04.701 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.711+0000 7f9bf77fe640 10 monclient(hunting): dump: 2024-03-29T10:52:04.701 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:04.701 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:04.701 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:04.701 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:04.701 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:04.701 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:04.701 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:04.701 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:04.701 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:04.701 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:04.701 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.711+0000 7f9bf77fe640 1 -- 172.21.15.112:0/2657891813 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1301116293 0 0) 0x7f9bfc004eb0 con 0x7f9c10142690 2024-03-29T10:52:04.701 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.711+0000 7f9bf77fe640 1 -- 172.21.15.112:0/2657891813 >> v1:172.21.15.118:6789/0 conn(0x7f9c100b17d0 legacy=0x7f9c100b5b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:52:04.701 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.712+0000 7f9bf77fe640 1 -- 172.21.15.112:0/2657891813 >> v1:172.21.15.112:6789/0 conn(0x7f9c101422b0 legacy=0x7f9c10071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:04.702 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.712+0000 7f9bf77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:52:04.702 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.712+0000 7f9bf77fe640 1 monclient: found mon.c 2024-03-29T10:52:04.702 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.712+0000 7f9bf77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:04.702 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.712+0000 7f9bf77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:52:04.702 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.712+0000 7f9bf77fe640 1 -- 172.21.15.112:0/2657891813 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1013b3e0 con 0x7f9c10142690 2024-03-29T10:52:04.702 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.712+0000 7f9bf77fe640 10 monclient: _finish_auth 0 2024-03-29T10:52:04.702 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.712+0000 7f9bf77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:52:04.702 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.712+0000 7f9bf77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:04.702 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.713+0000 7f9bf77fe640 1 -- 172.21.15.112:0/2657891813 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c08001fe0 con 0x7f9c10142690 2024-03-29T10:52:04.702 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.713+0000 7f9bf77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:04.702 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.713+0000 7f9bf77fe640 1 -- 172.21.15.112:0/2657891813 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9c08001e80 con 0x7f9c10142690 2024-03-29T10:52:04.702 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.713+0000 7f9bf77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:04.702 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.713+0000 7f9bf77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:04.704 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.713+0000 7f9bf77fe640 10 monclient: dump: 2024-03-29T10:52:04.704 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:04.704 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:04.704 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:04.704 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:04.704 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:04.704 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:04.704 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:04.704 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:04.705 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:04.705 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:04.705 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.713+0000 7f9c0d7fa640 4 set_mon_vals no callback set 2024-03-29T10:52:04.705 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.714+0000 7f9c154ff640 5 monclient: authenticate success, global_id 4256 2024-03-29T10:52:04.705 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.714+0000 7f9c154ff640 10 monclient: _renew_subs 2024-03-29T10:52:04.705 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.714+0000 7f9c154ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:52:04.705 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.714+0000 7f9c154ff640 1 -- 172.21.15.112:0/2657891813 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c1013b900 con 0x7f9c10142690 2024-03-29T10:52:04.705 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.714+0000 7f9c154ff640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:04.705 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.714+0000 7f9c154ff640 10 monclient: _renew_subs 2024-03-29T10:52:04.705 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.714+0000 7f9c154ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:52:04.705 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.714+0000 7f9c154ff640 1 -- 172.21.15.112:0/2657891813 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c1013b050 con 0x7f9c10142690 2024-03-29T10:52:04.705 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.714+0000 7f9c154ff640 1 librados: init done 2024-03-29T10:52:04.705 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.716+0000 7f9c154ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:04.705 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.717+0000 7f9c154ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:04.705 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.717+0000 7f9c154ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:52:04.706 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.717+0000 7f9c154ff640 1 -- 172.21.15.112:0/2657891813 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c10075f10 con 0x7f9c10142690 2024-03-29T10:52:04.707 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.718+0000 7f9bf77fe640 1 -- 172.21.15.112:0/2657891813 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f9c0801ac80 con 0x7f9c10142690 2024-03-29T10:52:04.707 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.722+0000 7f9c0effd640 10 monclient: get_auth_request con 0x7f9bfc06e150 auth_method 0 2024-03-29T10:52:04.707 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.722+0000 7f9bf77fe640 1 -- 172.21.15.112:0/2657891813 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f9c080548d0 con 0x7f9c10142690 2024-03-29T10:52:04.707 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.722+0000 7f9bf77fe640 10 client.4256.objecter ms_dispatch 0x7f9c10000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:04.707 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.723+0000 7f9bf77fe640 3 client.4256.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:04.708 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.723+0000 7f9bf77fe640 3 client.4256.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:04.708 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.723+0000 7f9bf77fe640 20 client.4256.objecter dump_active .. 0 homeless 2024-03-29T10:52:04.708 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.723+0000 7f9bf77fe640 10 client.4256.objecter ms_handle_connect 0x7f9bfc06e150 2024-03-29T10:52:04.708 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.727+0000 7f9bf77fe640 1 -- 172.21.15.112:0/2657891813 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9c080503e0 con 0x7f9c10142690 2024-03-29T10:52:04.708 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.727+0000 7f9bf77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:04.708 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:04.727+0000 7f9bf77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:05.033 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:05.051+0000 7f9c154ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:05.033 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:05.051+0000 7f9c154ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:05.033 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:05.051+0000 7f9c154ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:52:05.033 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:05.051+0000 7f9c154ff640 1 -- 172.21.15.112:0/2657891813 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c1013b570 con 0x7f9c10142690 2024-03-29T10:52:05.034 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:05.053+0000 7f9bf77fe640 1 -- 172.21.15.112:0/2657891813 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f9c080503e0 con 0x7f9c10142690 2024-03-29T10:52:05.034 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:05.053+0000 7f9bf77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:05.034 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:05.053+0000 7f9bf77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:05.034 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:52:05.035 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:05.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:05.059+0000 7f9c154ff640 10 librados: watch_flush enter 2024-03-29T10:52:05.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:05.059+0000 7f9c154ff640 10 librados: watch_flush exit 2024-03-29T10:52:05.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:05.059+0000 7f9c154ff640 20 client.4256.objecter shutdown clearing up homeless session... 2024-03-29T10:52:05.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:05.059+0000 7f9c154ff640 10 client.4256.objecter successfully canceled tick 2024-03-29T10:52:05.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:05.060+0000 7f9c154ff640 1 -- 172.21.15.112:0/2657891813 >> v1:172.21.15.118:6816/3181073408 conn(0x7f9bfc06e150 legacy=0x7f9bfc070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:05.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:05.060+0000 7f9c154ff640 10 monclient: shutdown 2024-03-29T10:52:05.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:05.060+0000 7f9c154ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:05.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:05.060+0000 7f9c154ff640 1 -- 172.21.15.112:0/2657891813 >> v1:172.21.15.112:6790/0 conn(0x7f9c10142690 legacy=0x7f9c101e6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:05.042 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:05.061+0000 7f9c154ff640 1 -- 172.21.15.112:0/2657891813 shutdown_connections 2024-03-29T10:52:05.042 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:05.061+0000 7f9c154ff640 1 -- 172.21.15.112:0/2657891813 >> 172.21.15.112:0/2657891813 conn(0x7f9c10066a00 msgr2=0x7f9c1013e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:05.042 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:05.061+0000 7f9c154ff640 1 -- 172.21.15.112:0/2657891813 shutdown_connections 2024-03-29T10:52:05.042 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:05.061+0000 7f9c154ff640 1 -- 172.21.15.112:0/2657891813 wait complete. 2024-03-29T10:52:05.042 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:05.061+0000 7f9c154ff640 1 librados: shutdown 2024-03-29T10:52:05.062 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:11.869 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:12.022 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.040+0000 7f16bd6e2640 10 monclient: get_monmap_and_config 2024-03-29T10:52:12.022 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.040+0000 7f16bd6e2640 10 monclient: build_initial_monmap 2024-03-29T10:52:12.023 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.040+0000 7f16bd6e2640 10 monclient: monmap: 2024-03-29T10:52:12.023 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:52:12.023 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:12.023 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:52:12.041668+0000 2024-03-29T10:52:12.023 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:52:12.041668+0000 2024-03-29T10:52:12.023 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:12.023 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:12.023 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:52:12.024 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:52:12.024 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:52:12.024 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:12.024 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.041+0000 7f16bd6e2640 1 Processor -- start 2024-03-29T10:52:12.025 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.041+0000 7f16bd6e2640 1 -- start start 2024-03-29T10:52:12.025 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.041+0000 7f16bd6e2640 10 monclient: init 2024-03-29T10:52:12.025 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.042+0000 7f16bd6e2640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:12.025 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.043+0000 7f16bd6e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:12.027 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.043+0000 7f16bd6e2640 10 monclient(hunting): picked mon.noname-b con 0x7f16b80a9ac0 addr v1:172.21.15.112:6790/0 2024-03-29T10:52:12.027 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.046+0000 7f16bd6e2640 10 monclient(hunting): picked mon.noname-c con 0x7f16b813ee90 addr v1:172.21.15.118:6789/0 2024-03-29T10:52:12.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.049+0000 7f16b6ffd640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f16b80a9ac0 0x7f16b80adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:34771/0 (socket says 172.21.15.112:34771) 2024-03-29T10:52:12.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.049+0000 7f16bd6e2640 10 monclient(hunting): picked mon.noname-a con 0x7f16b80af900 addr v1:172.21.15.112:6789/0 2024-03-29T10:52:12.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.049+0000 7f16bd6e2640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b80afd10 con 0x7f16b80af900 2024-03-29T10:52:12.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.049+0000 7f16bd6e2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b80af6f0 con 0x7f16b813ee90 2024-03-29T10:52:12.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.049+0000 7f16bd6e2640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b80aac80 con 0x7f16b80a9ac0 2024-03-29T10:52:12.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.049+0000 7f16bd6e2640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:12.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.049+0000 7f16bd6e2640 10 monclient(hunting): authenticate will time out at 887.353584s 2024-03-29T10:52:12.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.049+0000 7f16b6ffd640 1 -- 172.21.15.112:0/1409237098 learned_addr learned my addr 172.21.15.112:0/1409237098 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:52:12.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.050+0000 7f16b5ffb640 1 -- 172.21.15.112:0/1409237098 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109553888 0 0) 0x7f16b80afd10 con 0x7f16b80af900 2024-03-29T10:52:12.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.050+0000 7f16b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:12.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.050+0000 7f16b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:12.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.050+0000 7f16b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:12.033 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.050+0000 7f16b5ffb640 10 monclient(hunting): my global_id is 4260 2024-03-29T10:52:12.033 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.050+0000 7f16b5ffb640 1 -- 172.21.15.112:0/1409237098 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a0003520 con 0x7f16b80af900 2024-03-29T10:52:12.035 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.053+0000 7f16b5ffb640 1 -- 172.21.15.112:0/1409237098 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252526986 0 0) 0x7f16b80aac80 con 0x7f16b80a9ac0 2024-03-29T10:52:12.035 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.053+0000 7f16b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:12.035 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.053+0000 7f16b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:12.035 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.053+0000 7f16b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:12.035 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.053+0000 7f16b5ffb640 10 monclient(hunting): my global_id is 4259 2024-03-29T10:52:12.035 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.053+0000 7f16b5ffb640 1 -- 172.21.15.112:0/1409237098 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16b80afd10 con 0x7f16b80a9ac0 2024-03-29T10:52:12.035 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.053+0000 7f16b5ffb640 1 -- 172.21.15.112:0/1409237098 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915062509 0 0) 0x7f16b80af6f0 con 0x7f16b813ee90 2024-03-29T10:52:12.035 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.053+0000 7f16b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:12.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.053+0000 7f16b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:12.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.053+0000 7f16b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:12.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.053+0000 7f16b5ffb640 10 monclient(hunting): my global_id is 4267 2024-03-29T10:52:12.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.053+0000 7f16b5ffb640 1 -- 172.21.15.112:0/1409237098 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16b80aac80 con 0x7f16b813ee90 2024-03-29T10:52:12.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.053+0000 7f16b5ffb640 1 -- 172.21.15.112:0/1409237098 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4203105471 0 0) 0x7f16a0003520 con 0x7f16b80af900 2024-03-29T10:52:12.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.053+0000 7f16b5ffb640 1 -- 172.21.15.112:0/1409237098 >> v1:172.21.15.112:6790/0 conn(0x7f16b80a9ac0 legacy=0x7f16b80adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:12.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.053+0000 7f16b5ffb640 1 -- 172.21.15.112:0/1409237098 >> v1:172.21.15.118:6789/0 conn(0x7f16b813ee90 legacy=0x7f16b813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:12.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.053+0000 7f16b5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:52:12.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.053+0000 7f16b5ffb640 1 monclient: found mon.noname-a 2024-03-29T10:52:12.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.053+0000 7f16b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:12.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.053+0000 7f16b5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T10:52:12.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.053+0000 7f16b5ffb640 1 -- 172.21.15.112:0/1409237098 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b80aaf60 con 0x7f16b80af900 2024-03-29T10:52:12.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.054+0000 7f16b5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:52:12.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.054+0000 7f16b5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:52:12.037 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.054+0000 7f16b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:12.037 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.054+0000 7f16b5ffb640 1 -- 172.21.15.112:0/1409237098 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f16b0001990 con 0x7f16b80af900 2024-03-29T10:52:12.037 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.054+0000 7f16b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:12.037 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.054+0000 7f16b5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:12.037 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.054+0000 7f16b5ffb640 10 monclient: dump: 2024-03-29T10:52:12.037 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:12.037 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:12.037 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:12.037 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:12.037 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:12.037 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:12.037 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:12.037 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:12.037 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:12.037 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:12.038 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.054+0000 7f16bd6e2640 5 monclient: authenticate success, global_id 4260 2024-03-29T10:52:12.038 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.054+0000 7f16bd6e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:12.039 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.055+0000 7f16b5ffb640 1 -- 172.21.15.112:0/1409237098 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16b0001bb0 con 0x7f16b80af900 2024-03-29T10:52:12.039 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.055+0000 7f16b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:12.039 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.055+0000 7f16b5ffb640 1 -- 172.21.15.112:0/1409237098 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f16b0001f40 con 0x7f16b80af900 2024-03-29T10:52:12.039 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.055+0000 7f16bd6e2640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:12.039 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.055+0000 7f16bd6e2640 4 set_mon_vals no callback set 2024-03-29T10:52:12.039 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.056+0000 7f16bd6e2640 10 monclient: shutdown 2024-03-29T10:52:12.039 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.056+0000 7f16b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:12.039 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.056+0000 7f16b5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:12.039 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.056+0000 7f16b5ffb640 10 monclient: dump: 2024-03-29T10:52:12.039 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:12.039 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:12.039 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:12.039 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:12.040 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:12.040 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:12.040 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:12.040 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:12.040 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:12.040 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:12.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.056+0000 7f16bd6e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:12.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.056+0000 7f16bd6e2640 1 -- 172.21.15.112:0/1409237098 >> v1:172.21.15.112:6789/0 conn(0x7f16b80af900 legacy=0x7f16b8149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:12.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.056+0000 7f16bd6e2640 1 -- 172.21.15.112:0/1409237098 shutdown_connections 2024-03-29T10:52:12.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.056+0000 7f16bd6e2640 1 -- 172.21.15.112:0/1409237098 >> 172.21.15.112:0/1409237098 conn(0x7f16b80b35e0 msgr2=0x7f16b80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:12.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.057+0000 7f16bd6e2640 1 -- 172.21.15.112:0/1409237098 shutdown_connections 2024-03-29T10:52:12.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.057+0000 7f16bd6e2640 1 -- 172.21.15.112:0/1409237098 wait complete. 2024-03-29T10:52:12.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.058+0000 7f16bd6e2640 10 monclient: build_initial_monmap 2024-03-29T10:52:12.042 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.058+0000 7f16bd6e2640 10 monclient: monmap: 2024-03-29T10:52:12.042 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:52:12.042 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:12.042 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:52:12.042 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:52:12.042 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:12.042 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:12.042 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:52:12.042 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:52:12.042 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:52:12.042 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:12.042 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.058+0000 7f16bd6e2640 1 librados: starting msgr at 2024-03-29T10:52:12.043 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.058+0000 7f16bd6e2640 1 librados: starting objecter 2024-03-29T10:52:12.043 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.059+0000 7f16bd6e2640 1 Processor -- start 2024-03-29T10:52:12.043 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.059+0000 7f16bd6e2640 1 -- start start 2024-03-29T10:52:12.043 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.059+0000 7f16bd6e2640 1 librados: setting wanted keys 2024-03-29T10:52:12.043 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.059+0000 7f16bd6e2640 1 librados: calling monclient init 2024-03-29T10:52:12.043 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.059+0000 7f16bd6e2640 10 monclient: init 2024-03-29T10:52:12.043 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.060+0000 7f16bd6e2640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:12.043 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.060+0000 7f16bd6e2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:52:12.049 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.063+0000 7f16bd6e2640 10 monclient(hunting): picked mon.noname-a con 0x7f16b80a9ac0 addr v1:172.21.15.112:6789/0 2024-03-29T10:52:12.049 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.063+0000 7f16bd6e2640 10 monclient(hunting): picked mon.noname-b con 0x7f16b80af900 addr v1:172.21.15.118:6789/0 2024-03-29T10:52:12.049 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.063+0000 7f16bd6e2640 10 monclient(hunting): picked mon.noname-c con 0x7f16b813ee90 addr v1:172.21.15.112:6790/0 2024-03-29T10:52:12.049 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.063+0000 7f16bd6e2640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b8136bc0 con 0x7f16b80a9ac0 2024-03-29T10:52:12.049 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.063+0000 7f16bd6e2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b80b2d10 con 0x7f16b80af900 2024-03-29T10:52:12.049 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.063+0000 7f16bd6e2640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b80b3000 con 0x7f16b813ee90 2024-03-29T10:52:12.049 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.063+0000 7f16bd6e2640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:12.049 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.063+0000 7f16bd6e2640 10 monclient(hunting): authenticate will time out at 887.368163s 2024-03-29T10:52:12.055 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.068+0000 7f16b67fc640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f16b80af900 0x7f16b81358b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:52375/0 (socket says 172.21.15.112:52375) 2024-03-29T10:52:12.055 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.070+0000 7f16b6ffd640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f16b80a9ac0 0x7f16b813f8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:45151/0 (socket says 172.21.15.112:45151) 2024-03-29T10:52:12.055 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.070+0000 7f16b6ffd640 1 -- 172.21.15.112:0/3931681192 learned_addr learned my addr 172.21.15.112:0/3931681192 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:52:12.055 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.070+0000 7f169f7fe640 10 client.?.objecter ms_handle_connect 0x7f16b80a9ac0 2024-03-29T10:52:12.055 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.070+0000 7f169f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:12.055 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.070+0000 7f169f7fe640 1 -- 172.21.15.112:0/3931681192 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455808833 0 0) 0x7f16b8136bc0 con 0x7f16b80a9ac0 2024-03-29T10:52:12.056 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.071+0000 7f169f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:12.056 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.071+0000 7f169f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:12.056 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.071+0000 7f169f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:12.056 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.071+0000 7f169f7fe640 10 monclient(hunting): my global_id is 4263 2024-03-29T10:52:12.056 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.071+0000 7f169f7fe640 1 -- 172.21.15.112:0/3931681192 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a4004eb0 con 0x7f16b80a9ac0 2024-03-29T10:52:12.056 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.071+0000 7f169f7fe640 10 client.?.objecter ms_handle_connect 0x7f16b80af900 2024-03-29T10:52:12.056 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.071+0000 7f169f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:12.056 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.071+0000 7f169f7fe640 1 -- 172.21.15.112:0/3931681192 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 818746741 0 0) 0x7f16a4004eb0 con 0x7f16b80a9ac0 2024-03-29T10:52:12.056 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.071+0000 7f169f7fe640 1 -- 172.21.15.112:0/3931681192 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16b8136bc0 con 0x7f16b80a9ac0 2024-03-29T10:52:12.056 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.071+0000 7f169f7fe640 1 -- 172.21.15.112:0/3931681192 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f16ac001970 con 0x7f16b80a9ac0 2024-03-29T10:52:12.056 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.071+0000 7f169f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:12.056 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.071+0000 7f169f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:12.056 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.071+0000 7f169f7fe640 10 monclient(hunting): dump: 2024-03-29T10:52:12.056 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:12.057 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:12.057 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:12.057 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:12.057 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:12.057 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:12.057 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:12.057 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:12.057 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:12.057 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:12.057 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.071+0000 7f169f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:12.057 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.071+0000 7f169f7fe640 1 -- 172.21.15.112:0/3931681192 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089592045 0 0) 0x7f16b80b2d10 con 0x7f16b80af900 2024-03-29T10:52:12.057 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.071+0000 7f169f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:12.057 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.071+0000 7f169f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:12.057 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.071+0000 7f169f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:12.058 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.071+0000 7f169f7fe640 10 monclient(hunting): my global_id is 4270 2024-03-29T10:52:12.058 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.071+0000 7f169f7fe640 1 -- 172.21.15.112:0/3931681192 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a4004eb0 con 0x7f16b80af900 2024-03-29T10:52:12.058 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.072+0000 7f169f7fe640 1 -- 172.21.15.112:0/3931681192 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1604288671 0 0) 0x7f16b8136bc0 con 0x7f16b80a9ac0 2024-03-29T10:52:12.058 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.072+0000 7f169f7fe640 1 -- 172.21.15.112:0/3931681192 >> v1:172.21.15.112:6790/0 conn(0x7f16b813ee90 legacy=0x7f16b81de4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:52:12.058 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.072+0000 7f169f7fe640 1 -- 172.21.15.112:0/3931681192 >> v1:172.21.15.118:6789/0 conn(0x7f16b80af900 legacy=0x7f16b81358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:12.058 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.072+0000 7f169f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:52:12.058 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.072+0000 7f169f7fe640 1 monclient: found mon.a 2024-03-29T10:52:12.058 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.072+0000 7f169f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:12.058 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.072+0000 7f169f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:52:12.058 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.072+0000 7f169f7fe640 1 -- 172.21.15.112:0/3931681192 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b80b3190 con 0x7f16b80a9ac0 2024-03-29T10:52:12.058 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.072+0000 7f169f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:52:12.058 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.072+0000 7f169f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:52:12.058 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.072+0000 7f169f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:12.058 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.072+0000 7f16bd6e2640 5 monclient: authenticate success, global_id 4263 2024-03-29T10:52:12.059 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.072+0000 7f16bd6e2640 10 monclient: _renew_subs 2024-03-29T10:52:12.059 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.072+0000 7f16bd6e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:52:12.059 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.072+0000 7f16bd6e2640 1 -- 172.21.15.112:0/3931681192 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b8137f70 con 0x7f16b80a9ac0 2024-03-29T10:52:12.059 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.072+0000 7f16bd6e2640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:12.059 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.072+0000 7f16bd6e2640 10 monclient: _renew_subs 2024-03-29T10:52:12.059 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.072+0000 7f16bd6e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:52:12.059 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.072+0000 7f16bd6e2640 1 -- 172.21.15.112:0/3931681192 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b80b26f0 con 0x7f16b80a9ac0 2024-03-29T10:52:12.059 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.072+0000 7f16bd6e2640 1 librados: init done 2024-03-29T10:52:12.061 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.075+0000 7f169f7fe640 1 -- 172.21.15.112:0/3931681192 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16ac001d60 con 0x7f16b80a9ac0 2024-03-29T10:52:12.061 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.075+0000 7f169f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:12.061 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.075+0000 7f169f7fe640 1 -- 172.21.15.112:0/3931681192 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f16ac004c30 con 0x7f16b80a9ac0 2024-03-29T10:52:12.061 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.075+0000 7f169f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:12.061 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.075+0000 7f169f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:12.061 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.075+0000 7f169f7fe640 10 monclient: dump: 2024-03-29T10:52:12.061 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:12.062 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:12.062 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:12.062 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:12.062 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:12.062 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:12.062 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:12.062 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:12.062 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:12.062 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:12.062 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.075+0000 7f169f7fe640 1 -- 172.21.15.112:0/3931681192 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f16ac01ac00 con 0x7f16b80a9ac0 2024-03-29T10:52:12.062 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.076+0000 7f169f7fe640 1 -- 172.21.15.112:0/3931681192 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f16ac084c20 con 0x7f16b80a9ac0 2024-03-29T10:52:12.062 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.076+0000 7f169f7fe640 10 client.4263.objecter ms_dispatch 0x7f16b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:12.062 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.076+0000 7f169f7fe640 3 client.4263.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:12.063 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.076+0000 7f169f7fe640 3 client.4263.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:12.063 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.076+0000 7f169f7fe640 20 client.4263.objecter dump_active .. 0 homeless 2024-03-29T10:52:12.063 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.076+0000 7f16b67fc640 10 monclient: get_auth_request con 0x7f16a406ddb0 auth_method 0 2024-03-29T10:52:12.063 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.076+0000 7f169d7fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:12.063 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.076+0000 7f169d7fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:12.063 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.076+0000 7f169d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:52:12.063 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.076+0000 7f169d7fa640 1 -- 172.21.15.112:0/3931681192 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b80b3320 con 0x7f16b80a9ac0 2024-03-29T10:52:12.063 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.076+0000 7f16b57fa640 4 set_mon_vals no callback set 2024-03-29T10:52:12.063 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.077+0000 7f169f7fe640 10 client.4263.objecter ms_handle_connect 0x7f16a406ddb0 2024-03-29T10:52:12.067 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.086+0000 7f169f7fe640 1 -- 172.21.15.112:0/3931681192 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f16ac0503b0 con 0x7f16b80a9ac0 2024-03-29T10:52:12.067 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.086+0000 7f169f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:12.067 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.086+0000 7f169f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:12.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.407+0000 7f169d7fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:12.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.407+0000 7f169d7fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:12.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.407+0000 7f169d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:52:12.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.407+0000 7f169d7fa640 1 -- 172.21.15.112:0/3931681192 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16b813f590 con 0x7f16b80a9ac0 2024-03-29T10:52:12.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.412+0000 7f169f7fe640 1 -- 172.21.15.112:0/3931681192 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f16ac001b50 con 0x7f16b80a9ac0 2024-03-29T10:52:12.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.412+0000 7f169f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:12.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.413+0000 7f169f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:12.395 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:52:12.395 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:12.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.416+0000 7f169d7fa640 10 librados: watch_flush enter 2024-03-29T10:52:12.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.418+0000 7f169d7fa640 10 librados: watch_flush exit 2024-03-29T10:52:12.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.418+0000 7f169d7fa640 20 client.4263.objecter shutdown clearing up homeless session... 2024-03-29T10:52:12.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.418+0000 7f169d7fa640 10 client.4263.objecter successfully canceled tick 2024-03-29T10:52:12.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.418+0000 7f169d7fa640 1 -- 172.21.15.112:0/3931681192 >> v1:172.21.15.118:6816/3181073408 conn(0x7f16a406ddb0 legacy=0x7f16a4070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:12.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.418+0000 7f169d7fa640 10 monclient: shutdown 2024-03-29T10:52:12.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.418+0000 7f169d7fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:12.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.418+0000 7f169d7fa640 1 -- 172.21.15.112:0/3931681192 >> v1:172.21.15.112:6789/0 conn(0x7f16b80a9ac0 legacy=0x7f16b813f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:12.401 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.419+0000 7f169d7fa640 1 -- 172.21.15.112:0/3931681192 shutdown_connections 2024-03-29T10:52:12.401 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.419+0000 7f169d7fa640 1 -- 172.21.15.112:0/3931681192 >> 172.21.15.112:0/3931681192 conn(0x7f16b80b35e0 msgr2=0x7f16b813af70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:12.401 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.419+0000 7f169d7fa640 1 -- 172.21.15.112:0/3931681192 shutdown_connections 2024-03-29T10:52:12.401 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.419+0000 7f169d7fa640 1 -- 172.21.15.112:0/3931681192 wait complete. 2024-03-29T10:52:12.401 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:12.419+0000 7f169d7fa640 1 librados: shutdown 2024-03-29T10:52:12.419 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:19.226 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:19.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.395+0000 7feedee1b640 10 monclient: get_monmap_and_config 2024-03-29T10:52:19.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.395+0000 7feedee1b640 10 monclient: build_initial_monmap 2024-03-29T10:52:19.378 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.396+0000 7feedee1b640 10 monclient: monmap: 2024-03-29T10:52:19.378 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:52:19.378 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:19.378 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:52:19.396952+0000 2024-03-29T10:52:19.378 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:52:19.396952+0000 2024-03-29T10:52:19.378 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:19.378 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:19.378 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:52:19.378 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:52:19.378 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:52:19.378 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:19.379 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.398+0000 7feedee1b640 1 Processor -- start 2024-03-29T10:52:19.383 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.401+0000 7feedee1b640 1 -- start start 2024-03-29T10:52:19.385 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.401+0000 7feedee1b640 10 monclient: init 2024-03-29T10:52:19.385 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.403+0000 7feedee1b640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:19.385 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.403+0000 7feedee1b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:19.385 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.403+0000 7feedee1b640 10 monclient(hunting): picked mon.noname-b con 0x7feed0006b10 addr v1:172.21.15.112:6790/0 2024-03-29T10:52:19.385 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.403+0000 7feedee1b640 10 monclient(hunting): picked mon.noname-c con 0x7feed0006ef0 addr v1:172.21.15.118:6789/0 2024-03-29T10:52:19.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.403+0000 7feedee1b640 10 monclient(hunting): picked mon.noname-a con 0x7feed002efb0 addr v1:172.21.15.112:6789/0 2024-03-29T10:52:19.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.403+0000 7feedee1b640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed0008d80 con 0x7feed002efb0 2024-03-29T10:52:19.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.403+0000 7feedee1b640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed00302b0 con 0x7feed0006ef0 2024-03-29T10:52:19.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.403+0000 7feedee1b640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed0030800 con 0x7feed0006b10 2024-03-29T10:52:19.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.403+0000 7feedee1b640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:19.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.403+0000 7feedee1b640 10 monclient(hunting): authenticate will time out at 894.707298s 2024-03-29T10:52:19.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.407+0000 7feedde19640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7feed0006b10 0x7feed00c46e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37781/0 (socket says 172.21.15.112:37781) 2024-03-29T10:52:19.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.407+0000 7feedde19640 1 -- 172.21.15.112:0/1831641898 learned_addr learned my addr 172.21.15.112:0/1831641898 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:52:19.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.409+0000 7feedce17640 1 -- 172.21.15.112:0/1831641898 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 449054562 0 0) 0x7feed00302b0 con 0x7feed0006ef0 2024-03-29T10:52:19.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.409+0000 7feedce17640 10 monclient(hunting): _negotiate 2024-03-29T10:52:19.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.409+0000 7feedce17640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:19.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.409+0000 7feedce17640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:19.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.409+0000 7feedce17640 10 monclient(hunting): my global_id is 4273 2024-03-29T10:52:19.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.409+0000 7feedce17640 1 -- 172.21.15.112:0/1831641898 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feec4003520 con 0x7feed0006ef0 2024-03-29T10:52:19.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.409+0000 7feedce17640 1 -- 172.21.15.112:0/1831641898 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195114849 0 0) 0x7feed0008d80 con 0x7feed002efb0 2024-03-29T10:52:19.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.410+0000 7feedce17640 10 monclient(hunting): _negotiate 2024-03-29T10:52:19.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.410+0000 7feedce17640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:19.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.410+0000 7feedce17640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:19.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.410+0000 7feedce17640 10 monclient(hunting): my global_id is 4266 2024-03-29T10:52:19.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.410+0000 7feedce17640 1 -- 172.21.15.112:0/1831641898 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feed00302b0 con 0x7feed002efb0 2024-03-29T10:52:19.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.410+0000 7feedce17640 1 -- 172.21.15.112:0/1831641898 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3916091659 0 0) 0x7feec4003520 con 0x7feed0006ef0 2024-03-29T10:52:19.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.410+0000 7feedce17640 1 -- 172.21.15.112:0/1831641898 >> v1:172.21.15.112:6790/0 conn(0x7feed0006b10 legacy=0x7feed00c46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:19.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.410+0000 7feedce17640 1 -- 172.21.15.112:0/1831641898 >> v1:172.21.15.112:6789/0 conn(0x7feed002efb0 legacy=0x7feed00cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:19.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.410+0000 7feedce17640 10 monclient: _finish_hunting 0 2024-03-29T10:52:19.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.410+0000 7feedce17640 1 monclient: found mon.noname-c 2024-03-29T10:52:19.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.410+0000 7feedce17640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:19.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.410+0000 7feedce17640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T10:52:19.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.410+0000 7feedce17640 1 -- 172.21.15.112:0/1831641898 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed0008930 con 0x7feed0006ef0 2024-03-29T10:52:19.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.414+0000 7feedce17640 10 monclient: _finish_auth 0 2024-03-29T10:52:19.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.414+0000 7feedce17640 10 monclient: _check_auth_tickets 2024-03-29T10:52:19.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.414+0000 7feedce17640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:19.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.414+0000 7feedce17640 1 -- 172.21.15.112:0/1831641898 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7feed40015c0 con 0x7feed0006ef0 2024-03-29T10:52:19.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.414+0000 7feedee1b640 5 monclient: authenticate success, global_id 4273 2024-03-29T10:52:19.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.414+0000 7feedee1b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:19.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.414+0000 7feedce17640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:19.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.415+0000 7feedce17640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:19.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.415+0000 7feedce17640 10 monclient: dump: 2024-03-29T10:52:19.401 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:19.401 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:19.401 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:19.401 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:19.401 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:19.401 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:19.401 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:19.401 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:19.401 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:19.401 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:19.401 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.415+0000 7feedce17640 1 -- 172.21.15.112:0/1831641898 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feed4001850 con 0x7feed0006ef0 2024-03-29T10:52:19.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.415+0000 7feedee1b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:19.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.415+0000 7feedce17640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:19.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.415+0000 7feedce17640 1 -- 172.21.15.112:0/1831641898 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7feed4001be0 con 0x7feed0006ef0 2024-03-29T10:52:19.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.415+0000 7feedee1b640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:19.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.415+0000 7feedee1b640 4 set_mon_vals no callback set 2024-03-29T10:52:19.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.415+0000 7feedce17640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:19.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.415+0000 7feedce17640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:19.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.415+0000 7feedce17640 10 monclient: dump: 2024-03-29T10:52:19.404 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:19.404 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:19.404 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:19.404 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:19.404 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:19.404 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:19.404 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:19.404 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:19.404 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:19.405 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:19.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.416+0000 7feedee1b640 10 monclient: shutdown 2024-03-29T10:52:19.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.416+0000 7feedee1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:19.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.416+0000 7feedee1b640 1 -- 172.21.15.112:0/1831641898 >> v1:172.21.15.118:6789/0 conn(0x7feed0006ef0 legacy=0x7feed00caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:19.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.416+0000 7feedee1b640 1 -- 172.21.15.112:0/1831641898 shutdown_connections 2024-03-29T10:52:19.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.416+0000 7feedee1b640 1 -- 172.21.15.112:0/1831641898 >> 172.21.15.112:0/1831641898 conn(0x7feed0005c00 msgr2=0x7feed0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:19.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.417+0000 7feedee1b640 1 -- 172.21.15.112:0/1831641898 shutdown_connections 2024-03-29T10:52:19.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.417+0000 7feedee1b640 1 -- 172.21.15.112:0/1831641898 wait complete. 2024-03-29T10:52:19.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.418+0000 7feedee1b640 10 monclient: build_initial_monmap 2024-03-29T10:52:19.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.418+0000 7feedee1b640 10 monclient: monmap: 2024-03-29T10:52:19.408 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:52:19.408 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:19.408 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:52:19.408 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:52:19.408 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:19.408 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:19.408 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:52:19.408 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:52:19.409 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:52:19.409 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:19.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.418+0000 7feedee1b640 1 librados: starting msgr at 2024-03-29T10:52:19.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.418+0000 7feedee1b640 1 librados: starting objecter 2024-03-29T10:52:19.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.418+0000 7feedee1b640 1 Processor -- start 2024-03-29T10:52:19.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.418+0000 7feedee1b640 1 -- start start 2024-03-29T10:52:19.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.418+0000 7feedee1b640 1 librados: setting wanted keys 2024-03-29T10:52:19.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.418+0000 7feedee1b640 1 librados: calling monclient init 2024-03-29T10:52:19.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.418+0000 7feedee1b640 10 monclient: init 2024-03-29T10:52:19.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.419+0000 7feedee1b640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:19.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.419+0000 7feedee1b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:19.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.419+0000 7feedee1b640 10 monclient(hunting): picked mon.noname-b con 0x7feed0006b10 addr v1:172.21.15.118:6789/0 2024-03-29T10:52:19.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.422+0000 7feedde19640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7feed0006b10 0x7feed01611f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:41005/0 (socket says 172.21.15.112:41005) 2024-03-29T10:52:19.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.422+0000 7feedee1b640 10 monclient(hunting): picked mon.noname-a con 0x7feed0006ef0 addr v1:172.21.15.112:6789/0 2024-03-29T10:52:19.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.425+0000 7feedee1b640 10 monclient(hunting): picked mon.noname-c con 0x7feed002cb40 addr v1:172.21.15.112:6790/0 2024-03-29T10:52:19.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.425+0000 7feedde19640 1 -- 172.21.15.112:0/2041355556 learned_addr learned my addr 172.21.15.112:0/2041355556 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:52:19.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.425+0000 7feedee1b640 1 -- 172.21.15.112:0/2041355556 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed0160a80 con 0x7feed0006ef0 2024-03-29T10:52:19.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.425+0000 7feedee1b640 1 -- 172.21.15.112:0/2041355556 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed002dd60 con 0x7feed0006b10 2024-03-29T10:52:19.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.425+0000 7feedee1b640 1 -- 172.21.15.112:0/2041355556 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed015e670 con 0x7feed002cb40 2024-03-29T10:52:19.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.425+0000 7feedee1b640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:19.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.425+0000 7feedee1b640 10 monclient(hunting): authenticate will time out at 894.729491s 2024-03-29T10:52:19.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.425+0000 7feece7fc640 10 client.?.objecter ms_handle_connect 0x7feed0006b10 2024-03-29T10:52:19.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.425+0000 7feece7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:19.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.425+0000 7feece7fc640 10 client.?.objecter ms_handle_connect 0x7feed0006ef0 2024-03-29T10:52:19.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.425+0000 7feece7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:19.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.425+0000 7feece7fc640 1 -- 172.21.15.112:0/2041355556 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95145176 0 0) 0x7feed002dd60 con 0x7feed0006b10 2024-03-29T10:52:19.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.425+0000 7feece7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:19.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.425+0000 7feece7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:19.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.425+0000 7feece7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:19.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.425+0000 7feece7fc640 10 monclient(hunting): my global_id is 4276 2024-03-29T10:52:19.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.425+0000 7feece7fc640 1 -- 172.21.15.112:0/2041355556 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feebc004eb0 con 0x7feed0006b10 2024-03-29T10:52:19.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.426+0000 7feece7fc640 1 -- 172.21.15.112:0/2041355556 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268992038 0 0) 0x7feed0160a80 con 0x7feed0006ef0 2024-03-29T10:52:19.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.426+0000 7feece7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:19.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.426+0000 7feece7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:19.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.426+0000 7feece7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:19.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.426+0000 7feece7fc640 10 monclient(hunting): my global_id is 4269 2024-03-29T10:52:19.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.426+0000 7feece7fc640 1 -- 172.21.15.112:0/2041355556 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feed002dd60 con 0x7feed0006ef0 2024-03-29T10:52:19.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.426+0000 7feece7fc640 1 -- 172.21.15.112:0/2041355556 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2800798704 0 0) 0x7feebc004eb0 con 0x7feed0006b10 2024-03-29T10:52:19.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.426+0000 7feece7fc640 1 -- 172.21.15.112:0/2041355556 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feed0160a80 con 0x7feed0006b10 2024-03-29T10:52:19.413 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.426+0000 7feece7fc640 1 -- 172.21.15.112:0/2041355556 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7feec80019d0 con 0x7feed0006b10 2024-03-29T10:52:19.413 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.426+0000 7feece7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:19.413 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.426+0000 7feece7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:19.413 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.426+0000 7feece7fc640 10 monclient(hunting): dump: 2024-03-29T10:52:19.413 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:19.413 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:19.413 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:19.413 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:19.413 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:19.416 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:19.416 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:19.416 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:19.416 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:19.416 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:19.421 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.426+0000 7feece7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:19.426 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.427+0000 7feece7fc640 1 -- 172.21.15.112:0/2041355556 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2149737464 0 0) 0x7feed0160a80 con 0x7feed0006b10 2024-03-29T10:52:19.426 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.427+0000 7feece7fc640 1 -- 172.21.15.112:0/2041355556 >> v1:172.21.15.112:6790/0 conn(0x7feed002cb40 legacy=0x7feed016b2b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:52:19.426 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.431+0000 7feece7fc640 1 -- 172.21.15.112:0/2041355556 >> v1:172.21.15.112:6789/0 conn(0x7feed0006ef0 legacy=0x7feed002c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:19.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.431+0000 7feece7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:52:19.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.431+0000 7feece7fc640 1 monclient: found mon.b 2024-03-29T10:52:19.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.431+0000 7feece7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:19.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.431+0000 7feece7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:52:19.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.431+0000 7feece7fc640 1 -- 172.21.15.112:0/2041355556 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed0161900 con 0x7feed0006b10 2024-03-29T10:52:19.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.431+0000 7feece7fc640 10 monclient: _finish_auth 0 2024-03-29T10:52:19.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.431+0000 7feece7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:52:19.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.431+0000 7feece7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:19.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.431+0000 7feedee1b640 5 monclient: authenticate success, global_id 4276 2024-03-29T10:52:19.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.431+0000 7feedee1b640 10 monclient: _renew_subs 2024-03-29T10:52:19.428 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.431+0000 7feedee1b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:52:19.428 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.431+0000 7feedee1b640 1 -- 172.21.15.112:0/2041355556 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed0160cd0 con 0x7feed0006b10 2024-03-29T10:52:19.428 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.431+0000 7feedee1b640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:19.428 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.431+0000 7feedee1b640 10 monclient: _renew_subs 2024-03-29T10:52:19.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.431+0000 7feedee1b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:52:19.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.431+0000 7feedee1b640 1 -- 172.21.15.112:0/2041355556 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feed016c9d0 con 0x7feed0006b10 2024-03-29T10:52:19.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.431+0000 7feedee1b640 1 librados: init done 2024-03-29T10:52:19.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.433+0000 7feece7fc640 1 -- 172.21.15.112:0/2041355556 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feec80047e0 con 0x7feed0006b10 2024-03-29T10:52:19.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.433+0000 7feece7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:19.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.433+0000 7feecffff640 4 set_mon_vals no callback set 2024-03-29T10:52:19.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.434+0000 7feedee1b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:19.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.434+0000 7feece7fc640 1 -- 172.21.15.112:0/2041355556 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7feec8004af0 con 0x7feed0006b10 2024-03-29T10:52:19.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.434+0000 7feece7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:19.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.434+0000 7feece7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:19.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.434+0000 7feece7fc640 10 monclient: dump: 2024-03-29T10:52:19.430 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:19.430 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:19.430 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:19.430 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:19.430 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:19.430 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:19.430 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:19.430 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:19.430 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:19.430 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:19.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.434+0000 7feece7fc640 1 -- 172.21.15.112:0/2041355556 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7feec801ab40 con 0x7feed0006b10 2024-03-29T10:52:19.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.435+0000 7feece7fc640 1 -- 172.21.15.112:0/2041355556 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7feec8084b70 con 0x7feed0006b10 2024-03-29T10:52:19.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.435+0000 7feece7fc640 10 client.4276.objecter ms_dispatch 0x7feed8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:19.431 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.435+0000 7feece7fc640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:19.431 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.435+0000 7feece7fc640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:19.431 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.435+0000 7feece7fc640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-29T10:52:19.431 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.435+0000 7feedee1b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:19.431 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.435+0000 7feedee1b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:52:19.431 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.435+0000 7feedee1b640 1 -- 172.21.15.112:0/2041355556 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feec0005080 con 0x7feed0006b10 2024-03-29T10:52:19.431 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.436+0000 7feedd618640 10 monclient: get_auth_request con 0x7feebc06db80 auth_method 0 2024-03-29T10:52:19.431 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.439+0000 7feece7fc640 10 client.4276.objecter ms_handle_connect 0x7feebc06db80 2024-03-29T10:52:19.431 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.443+0000 7feece7fc640 1 -- 172.21.15.112:0/2041355556 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feec8050300 con 0x7feed0006b10 2024-03-29T10:52:19.431 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.443+0000 7feece7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:19.431 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.443+0000 7feece7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:19.791 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.810+0000 7feedee1b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:19.792 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.810+0000 7feedee1b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:19.792 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.810+0000 7feedee1b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:52:19.792 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.810+0000 7feedee1b640 1 -- 172.21.15.112:0/2041355556 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feec0005330 con 0x7feed0006b10 2024-03-29T10:52:19.792 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.811+0000 7feece7fc640 1 -- 172.21.15.112:0/2041355556 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7feec8052400 con 0x7feed0006b10 2024-03-29T10:52:19.792 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.811+0000 7feece7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:19.792 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.811+0000 7feece7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:19.793 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:52:19.793 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:19.796 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.814+0000 7feedee1b640 10 librados: watch_flush enter 2024-03-29T10:52:19.796 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.815+0000 7feedee1b640 10 librados: watch_flush exit 2024-03-29T10:52:19.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.815+0000 7feedee1b640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-29T10:52:19.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.815+0000 7feedee1b640 10 client.4276.objecter successfully canceled tick 2024-03-29T10:52:19.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.815+0000 7feedee1b640 1 -- 172.21.15.112:0/2041355556 >> v1:172.21.15.118:6816/3181073408 conn(0x7feebc06db80 legacy=0x7feebc070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:19.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.815+0000 7feedee1b640 10 monclient: shutdown 2024-03-29T10:52:19.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.815+0000 7feedee1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:19.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.815+0000 7feedee1b640 1 -- 172.21.15.112:0/2041355556 >> v1:172.21.15.118:6789/0 conn(0x7feed0006b10 legacy=0x7feed01611f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:19.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.815+0000 7feedee1b640 1 -- 172.21.15.112:0/2041355556 shutdown_connections 2024-03-29T10:52:19.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.815+0000 7feedee1b640 1 -- 172.21.15.112:0/2041355556 >> 172.21.15.112:0/2041355556 conn(0x7feed0005c00 msgr2=0x7feed00d7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:19.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.815+0000 7feedee1b640 1 -- 172.21.15.112:0/2041355556 shutdown_connections 2024-03-29T10:52:19.799 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.816+0000 7feedee1b640 1 -- 172.21.15.112:0/2041355556 wait complete. 2024-03-29T10:52:19.799 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:19.816+0000 7feedee1b640 1 librados: shutdown 2024-03-29T10:52:19.813 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:26.619 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:26.782 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.801+0000 7f3f8984a640 10 monclient: get_monmap_and_config 2024-03-29T10:52:26.782 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.801+0000 7f3f8984a640 10 monclient: build_initial_monmap 2024-03-29T10:52:26.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.801+0000 7f3f8984a640 10 monclient: monmap: 2024-03-29T10:52:26.783 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:52:26.783 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:26.783 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:52:26.802300+0000 2024-03-29T10:52:26.783 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:52:26.802300+0000 2024-03-29T10:52:26.783 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:26.783 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:26.783 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:52:26.783 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:52:26.783 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:52:26.783 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:26.784 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.802+0000 7f3f8984a640 1 Processor -- start 2024-03-29T10:52:26.785 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.802+0000 7f3f8984a640 1 -- start start 2024-03-29T10:52:26.785 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.802+0000 7f3f8984a640 10 monclient: init 2024-03-29T10:52:26.785 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.803+0000 7f3f8984a640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:26.785 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.803+0000 7f3f8984a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:26.785 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.803+0000 7f3f8984a640 10 monclient(hunting): picked mon.noname-b con 0x7f3f840b0770 addr v1:172.21.15.112:6790/0 2024-03-29T10:52:26.785 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.803+0000 7f3f8984a640 10 monclient(hunting): picked mon.noname-c con 0x7f3f8414a700 addr v1:172.21.15.118:6789/0 2024-03-29T10:52:26.785 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.804+0000 7f3f8984a640 10 monclient(hunting): picked mon.noname-a con 0x7f3f840ac5a0 addr v1:172.21.15.112:6789/0 2024-03-29T10:52:26.785 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.804+0000 7f3f8984a640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f840aba70 con 0x7f3f840ac5a0 2024-03-29T10:52:26.786 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.804+0000 7f3f8984a640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f840aecb0 con 0x7f3f8414a700 2024-03-29T10:52:26.786 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.804+0000 7f3f8984a640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f840af150 con 0x7f3f840b0770 2024-03-29T10:52:26.786 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.804+0000 7f3f8984a640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:26.786 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.804+0000 7f3f8984a640 10 monclient(hunting): authenticate will time out at 902.108262s 2024-03-29T10:52:26.786 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.804+0000 7f3f837fe640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f3f840ac5a0 0x7f3f84151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46632/0 (socket says 172.21.15.112:46632) 2024-03-29T10:52:26.786 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.804+0000 7f3f837fe640 1 -- 172.21.15.112:0/530607730 learned_addr learned my addr 172.21.15.112:0/530607730 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:52:26.789 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.804+0000 7f3f81ffb640 1 -- 172.21.15.112:0/530607730 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620336174 0 0) 0x7f3f840af150 con 0x7f3f840b0770 2024-03-29T10:52:26.790 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:26.790 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:26.790 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:26.790 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 10 monclient(hunting): my global_id is 4265 2024-03-29T10:52:26.790 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 1 -- 172.21.15.112:0/530607730 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f6c003520 con 0x7f3f840b0770 2024-03-29T10:52:26.790 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 1 -- 172.21.15.112:0/530607730 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540441299 0 0) 0x7f3f840aba70 con 0x7f3f840ac5a0 2024-03-29T10:52:26.790 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:26.790 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:26.790 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:26.790 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 10 monclient(hunting): my global_id is 4272 2024-03-29T10:52:26.790 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 1 -- 172.21.15.112:0/530607730 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f840af150 con 0x7f3f840ac5a0 2024-03-29T10:52:26.790 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 1 -- 172.21.15.112:0/530607730 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2287937911 0 0) 0x7f3f840aecb0 con 0x7f3f8414a700 2024-03-29T10:52:26.791 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:26.791 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:26.791 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:26.791 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 10 monclient(hunting): my global_id is 4279 2024-03-29T10:52:26.791 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 1 -- 172.21.15.112:0/530607730 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f840aba70 con 0x7f3f8414a700 2024-03-29T10:52:26.791 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 1 -- 172.21.15.112:0/530607730 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1738035041 0 0) 0x7f3f6c003520 con 0x7f3f840b0770 2024-03-29T10:52:26.791 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 1 -- 172.21.15.112:0/530607730 >> v1:172.21.15.118:6789/0 conn(0x7f3f8414a700 legacy=0x7f3f840abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:26.791 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 1 -- 172.21.15.112:0/530607730 >> v1:172.21.15.112:6789/0 conn(0x7f3f840ac5a0 legacy=0x7f3f84151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:26.791 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:52:26.791 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 1 monclient: found mon.noname-b 2024-03-29T10:52:26.791 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:26.791 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:52:26.791 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 1 -- 172.21.15.112:0/530607730 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f840ac9d0 con 0x7f3f840b0770 2024-03-29T10:52:26.791 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 10 monclient: _finish_auth 0 2024-03-29T10:52:26.792 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:52:26.792 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:26.792 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f81ffb640 1 -- 172.21.15.112:0/530607730 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f3f7c001a10 con 0x7f3f840b0770 2024-03-29T10:52:26.792 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f8984a640 5 monclient: authenticate success, global_id 4265 2024-03-29T10:52:26.792 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.805+0000 7f3f8984a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:26.792 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.806+0000 7f3f81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:26.792 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.806+0000 7f3f81ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:26.792 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.806+0000 7f3f81ffb640 10 monclient: dump: 2024-03-29T10:52:26.792 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:26.792 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:26.792 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:26.792 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:26.792 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:26.792 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:26.792 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:26.793 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:26.793 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:26.793 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:26.793 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.806+0000 7f3f8984a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:26.793 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.806+0000 7f3f81ffb640 1 -- 172.21.15.112:0/530607730 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f7c001bf0 con 0x7f3f840b0770 2024-03-29T10:52:26.793 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.806+0000 7f3f81ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:26.793 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.806+0000 7f3f81ffb640 1 -- 172.21.15.112:0/530607730 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f3f7c002000 con 0x7f3f840b0770 2024-03-29T10:52:26.793 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.806+0000 7f3f81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:26.793 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.806+0000 7f3f81ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:26.793 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.806+0000 7f3f81ffb640 10 monclient: dump: 2024-03-29T10:52:26.793 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:26.793 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:26.793 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:26.793 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:26.793 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:26.794 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:26.794 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:26.794 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:26.794 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:26.794 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:26.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.806+0000 7f3f8984a640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:26.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.806+0000 7f3f8984a640 4 set_mon_vals no callback set 2024-03-29T10:52:26.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.807+0000 7f3f8984a640 10 monclient: shutdown 2024-03-29T10:52:26.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.807+0000 7f3f8984a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:26.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.807+0000 7f3f8984a640 1 -- 172.21.15.112:0/530607730 >> v1:172.21.15.112:6790/0 conn(0x7f3f840b0770 legacy=0x7f3f84149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:26.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.807+0000 7f3f8984a640 1 -- 172.21.15.112:0/530607730 shutdown_connections 2024-03-29T10:52:26.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.807+0000 7f3f8984a640 1 -- 172.21.15.112:0/530607730 >> 172.21.15.112:0/530607730 conn(0x7f3f8406b660 msgr2=0x7f3f8406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:26.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.807+0000 7f3f8984a640 1 -- 172.21.15.112:0/530607730 shutdown_connections 2024-03-29T10:52:26.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.807+0000 7f3f8984a640 1 -- 172.21.15.112:0/530607730 wait complete. 2024-03-29T10:52:26.795 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.808+0000 7f3f8984a640 10 monclient: build_initial_monmap 2024-03-29T10:52:26.795 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.808+0000 7f3f8984a640 10 monclient: monmap: 2024-03-29T10:52:26.795 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:52:26.795 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:26.795 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:52:26.795 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:52:26.795 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:26.795 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:26.795 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:52:26.795 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:52:26.795 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:52:26.795 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:26.795 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.808+0000 7f3f8984a640 1 librados: starting msgr at 2024-03-29T10:52:26.800 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.808+0000 7f3f8984a640 1 librados: starting objecter 2024-03-29T10:52:26.800 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.808+0000 7f3f8984a640 1 Processor -- start 2024-03-29T10:52:26.800 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.809+0000 7f3f8984a640 1 -- start start 2024-03-29T10:52:26.800 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.809+0000 7f3f8984a640 1 librados: setting wanted keys 2024-03-29T10:52:26.800 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.809+0000 7f3f8984a640 1 librados: calling monclient init 2024-03-29T10:52:26.800 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.809+0000 7f3f8984a640 10 monclient: init 2024-03-29T10:52:26.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.810+0000 7f3f8984a640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:26.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.810+0000 7f3f8984a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:26.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.810+0000 7f3f8984a640 10 monclient(hunting): picked mon.noname-c con 0x7f3f840ac5a0 addr v1:172.21.15.112:6790/0 2024-03-29T10:52:26.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.810+0000 7f3f8984a640 10 monclient(hunting): picked mon.noname-b con 0x7f3f840b0770 addr v1:172.21.15.118:6789/0 2024-03-29T10:52:26.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.810+0000 7f3f8984a640 10 monclient(hunting): picked mon.noname-a con 0x7f3f8414a700 addr v1:172.21.15.112:6789/0 2024-03-29T10:52:26.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.810+0000 7f3f8984a640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f841db6c0 con 0x7f3f8414a700 2024-03-29T10:52:26.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.810+0000 7f3f8984a640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f84123a80 con 0x7f3f840b0770 2024-03-29T10:52:26.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.810+0000 7f3f8984a640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f841468a0 con 0x7f3f840ac5a0 2024-03-29T10:52:26.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.810+0000 7f3f8984a640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:26.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.810+0000 7f3f8984a640 10 monclient(hunting): authenticate will time out at 902.114422s 2024-03-29T10:52:26.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.810+0000 7f3f827fc640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f3f840b0770 0x7f3f84145b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:35210/0 (socket says 172.21.15.112:35210) 2024-03-29T10:52:26.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.810+0000 7f3f827fc640 1 -- 172.21.15.112:0/2716650061 learned_addr learned my addr 172.21.15.112:0/2716650061 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:52:26.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.810+0000 7f3f6bfff640 10 client.?.objecter ms_handle_connect 0x7f3f840b0770 2024-03-29T10:52:26.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.811+0000 7f3f6bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:26.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.811+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3357327598 0 0) 0x7f3f84123a80 con 0x7f3f840b0770 2024-03-29T10:52:26.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.811+0000 7f3f6bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:26.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.811+0000 7f3f6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:26.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.811+0000 7f3f6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:26.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.811+0000 7f3f6bfff640 10 monclient(hunting): my global_id is 4282 2024-03-29T10:52:26.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.811+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f70004eb0 con 0x7f3f840b0770 2024-03-29T10:52:26.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.811+0000 7f3f6bfff640 10 client.?.objecter ms_handle_connect 0x7f3f840ac5a0 2024-03-29T10:52:26.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.811+0000 7f3f6bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:26.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.811+0000 7f3f6bfff640 10 client.?.objecter ms_handle_connect 0x7f3f8414a700 2024-03-29T10:52:26.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.811+0000 7f3f6bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:26.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.812+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1337727283 0 0) 0x7f3f841468a0 con 0x7f3f840ac5a0 2024-03-29T10:52:26.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.812+0000 7f3f6bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:26.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.812+0000 7f3f6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:26.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.812+0000 7f3f6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:26.803 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.812+0000 7f3f6bfff640 10 monclient(hunting): my global_id is 4268 2024-03-29T10:52:26.803 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.812+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f84123a80 con 0x7f3f840ac5a0 2024-03-29T10:52:26.803 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.812+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1918959340 0 0) 0x7f3f70004eb0 con 0x7f3f840b0770 2024-03-29T10:52:26.803 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.812+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f841468a0 con 0x7f3f840b0770 2024-03-29T10:52:26.803 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.812+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f3f78001950 con 0x7f3f840b0770 2024-03-29T10:52:26.803 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.812+0000 7f3f6bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:26.803 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.812+0000 7f3f6bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:26.803 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.812+0000 7f3f6bfff640 10 monclient(hunting): dump: 2024-03-29T10:52:26.803 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:26.803 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:26.803 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:26.803 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:26.803 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:26.803 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:26.804 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:26.804 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:26.804 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:26.804 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:26.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.812+0000 7f3f6bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:26.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.812+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 412816276 0 0) 0x7f3f841db6c0 con 0x7f3f8414a700 2024-03-29T10:52:26.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.812+0000 7f3f6bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:26.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.812+0000 7f3f6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:26.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.812+0000 7f3f6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:26.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.812+0000 7f3f6bfff640 10 monclient(hunting): my global_id is 4275 2024-03-29T10:52:26.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.812+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f70004eb0 con 0x7f3f8414a700 2024-03-29T10:52:26.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.813+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 107615978 0 0) 0x7f3f84123a80 con 0x7f3f840ac5a0 2024-03-29T10:52:26.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.813+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f841db6c0 con 0x7f3f840ac5a0 2024-03-29T10:52:26.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.813+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f3f7c001f00 con 0x7f3f840ac5a0 2024-03-29T10:52:26.805 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.813+0000 7f3f6bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:26.805 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.813+0000 7f3f6bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:26.805 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.813+0000 7f3f6bfff640 10 monclient(hunting): dump: 2024-03-29T10:52:26.805 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:26.805 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:26.805 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:26.805 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:26.805 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:26.805 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:26.805 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:26.805 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:26.805 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:26.805 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:26.805 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.813+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3080698839 0 0) 0x7f3f70004eb0 con 0x7f3f8414a700 2024-03-29T10:52:26.806 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f84123a80 con 0x7f3f8414a700 2024-03-29T10:52:26.806 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f3f740016d0 con 0x7f3f8414a700 2024-03-29T10:52:26.806 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f6bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:26.806 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f6bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:26.806 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f6bfff640 10 monclient(hunting): dump: 2024-03-29T10:52:26.806 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:26.806 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:26.806 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:26.806 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:26.806 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:26.806 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:26.806 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:26.806 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:26.807 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:26.807 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:26.807 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3333482648 0 0) 0x7f3f841468a0 con 0x7f3f840b0770 2024-03-29T10:52:26.807 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 >> v1:172.21.15.112:6790/0 conn(0x7f3f840ac5a0 legacy=0x7f3f841dbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:26.807 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 >> v1:172.21.15.112:6789/0 conn(0x7f3f8414a700 legacy=0x7f3f841244f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:26.807 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f6bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:52:26.807 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f6bfff640 1 monclient: found mon.b 2024-03-29T10:52:26.807 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:26.807 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f6bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:52:26.807 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f84146a30 con 0x7f3f840b0770 2024-03-29T10:52:26.807 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f6bfff640 10 monclient: _finish_auth 0 2024-03-29T10:52:26.807 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f6bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:52:26.807 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:26.807 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f8984a640 5 monclient: authenticate success, global_id 4282 2024-03-29T10:52:26.807 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f8984a640 10 monclient: _renew_subs 2024-03-29T10:52:26.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f8984a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:52:26.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f8984a640 1 -- 172.21.15.112:0/2716650061 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f841dc2f0 con 0x7f3f840b0770 2024-03-29T10:52:26.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f8984a640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:26.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f8984a640 10 monclient: _renew_subs 2024-03-29T10:52:26.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f8984a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:52:26.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f8984a640 1 -- 172.21.15.112:0/2716650061 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f84146480 con 0x7f3f840b0770 2024-03-29T10:52:26.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.814+0000 7f3f8984a640 1 librados: init done 2024-03-29T10:52:26.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.815+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f78003610 con 0x7f3f840b0770 2024-03-29T10:52:26.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.815+0000 7f3f6bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:26.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.815+0000 7f3f817fa640 4 set_mon_vals no callback set 2024-03-29T10:52:26.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.816+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f3f780039e0 con 0x7f3f840b0770 2024-03-29T10:52:26.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.816+0000 7f3f6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:26.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.816+0000 7f3f6bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:26.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.816+0000 7f3f6bfff640 10 monclient: dump: 2024-03-29T10:52:26.809 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:26.809 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:26.809 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:26.809 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:26.809 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:26.809 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:26.809 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:26.809 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:26.809 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:26.809 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:26.809 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.817+0000 7f3f8984a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:26.809 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.817+0000 7f3f8984a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:26.809 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.817+0000 7f3f8984a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:52:26.810 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.817+0000 7f3f8984a640 1 -- 172.21.15.112:0/2716650061 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f84125fa0 con 0x7f3f840b0770 2024-03-29T10:52:26.810 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.819+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f3f78003bc0 con 0x7f3f840b0770 2024-03-29T10:52:26.810 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.820+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f3f78084950 con 0x7f3f840b0770 2024-03-29T10:52:26.811 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.820+0000 7f3f6bfff640 10 client.4282.objecter ms_dispatch 0x7f3f84000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:26.811 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.820+0000 7f3f6bfff640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:26.811 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.820+0000 7f3f6bfff640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:26.811 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.820+0000 7f3f6bfff640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-29T10:52:26.811 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.820+0000 7f3f82ffd640 10 monclient: get_auth_request con 0x7f3f7006e100 auth_method 0 2024-03-29T10:52:26.811 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.821+0000 7f3f6bfff640 10 client.4282.objecter ms_handle_connect 0x7f3f7006e100 2024-03-29T10:52:26.811 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.827+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3f78050370 con 0x7f3f840b0770 2024-03-29T10:52:26.811 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.827+0000 7f3f6bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:26.811 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:26.827+0000 7f3f6bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:27.201 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:27.220+0000 7f3f8984a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:27.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:27.220+0000 7f3f8984a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:27.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:27.220+0000 7f3f8984a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:52:27.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:27.220+0000 7f3f8984a640 1 -- 172.21.15.112:0/2716650061 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f840ab560 con 0x7f3f840b0770 2024-03-29T10:52:27.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:27.221+0000 7f3f6bfff640 1 -- 172.21.15.112:0/2716650061 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f3f78050370 con 0x7f3f840b0770 2024-03-29T10:52:27.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:27.221+0000 7f3f6bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:27.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:27.221+0000 7f3f6bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:27.203 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:52:27.203 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:27.207 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:27.225+0000 7f3f8984a640 10 librados: watch_flush enter 2024-03-29T10:52:27.207 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:27.225+0000 7f3f8984a640 10 librados: watch_flush exit 2024-03-29T10:52:27.207 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:27.225+0000 7f3f8984a640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-29T10:52:27.207 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:27.225+0000 7f3f8984a640 10 client.4282.objecter successfully canceled tick 2024-03-29T10:52:27.207 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:27.225+0000 7f3f8984a640 1 -- 172.21.15.112:0/2716650061 >> v1:172.21.15.118:6816/3181073408 conn(0x7f3f7006e100 legacy=0x7f3f700705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:27.207 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:27.225+0000 7f3f8984a640 10 monclient: shutdown 2024-03-29T10:52:27.207 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:27.225+0000 7f3f8984a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:27.207 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:27.225+0000 7f3f8984a640 1 -- 172.21.15.112:0/2716650061 >> v1:172.21.15.118:6789/0 conn(0x7f3f840b0770 legacy=0x7f3f84145b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:27.207 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:27.225+0000 7f3f8984a640 1 -- 172.21.15.112:0/2716650061 shutdown_connections 2024-03-29T10:52:27.207 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:27.225+0000 7f3f8984a640 1 -- 172.21.15.112:0/2716650061 >> 172.21.15.112:0/2716650061 conn(0x7f3f8406b660 msgr2=0x7f3f840b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:27.208 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:27.226+0000 7f3f8984a640 1 -- 172.21.15.112:0/2716650061 shutdown_connections 2024-03-29T10:52:27.208 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:27.226+0000 7f3f8984a640 1 -- 172.21.15.112:0/2716650061 wait complete. 2024-03-29T10:52:27.208 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:27.227+0000 7f3f8984a640 1 librados: shutdown 2024-03-29T10:52:27.225 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:34.031 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:34.166 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.183+0000 7f364b0a8640 10 monclient: get_monmap_and_config 2024-03-29T10:52:34.166 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.183+0000 7f364b0a8640 10 monclient: build_initial_monmap 2024-03-29T10:52:34.166 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.183+0000 7f364b0a8640 10 monclient: monmap: 2024-03-29T10:52:34.166 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:52:34.166 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:34.166 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:52:34.185816+0000 2024-03-29T10:52:34.166 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:52:34.185816+0000 2024-03-29T10:52:34.167 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:34.167 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:34.167 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:52:34.167 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:52:34.167 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:52:34.167 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:34.168 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.185+0000 7f364b0a8640 1 Processor -- start 2024-03-29T10:52:34.168 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.185+0000 7f364b0a8640 1 -- start start 2024-03-29T10:52:34.168 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.185+0000 7f364b0a8640 10 monclient: init 2024-03-29T10:52:34.168 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.185+0000 7f364b0a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:34.168 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.186+0000 7f364b0a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:52:34.168 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.186+0000 7f364b0a8640 10 monclient(hunting): picked mon.noname-c con 0x7f36440b46b0 addr v1:172.21.15.118:6789/0 2024-03-29T10:52:34.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.186+0000 7f364b0a8640 10 monclient(hunting): picked mon.noname-b con 0x7f36440aeae0 addr v1:172.21.15.112:6790/0 2024-03-29T10:52:34.174 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.186+0000 7f3643fff640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f36440aeae0 0x7f364413f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59918/0 (socket says 172.21.15.112:59918) 2024-03-29T10:52:34.174 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.187+0000 7f3648e44640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f36440b46b0 0x7f36440b3b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:52176/0 (socket says 172.21.15.112:52176) 2024-03-29T10:52:34.174 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.187+0000 7f3643fff640 1 -- 172.21.15.112:0/1715241349 learned_addr learned my addr 172.21.15.112:0/1715241349 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:52:34.174 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.187+0000 7f364b0a8640 10 monclient(hunting): picked mon.noname-a con 0x7f364413fa10 addr v1:172.21.15.112:6789/0 2024-03-29T10:52:34.174 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.187+0000 7f364b0a8640 1 -- 172.21.15.112:0/1715241349 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36440b06a0 con 0x7f364413fa10 2024-03-29T10:52:34.174 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.187+0000 7f364b0a8640 1 -- 172.21.15.112:0/1715241349 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36440b4230 con 0x7f36440b46b0 2024-03-29T10:52:34.174 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.187+0000 7f364b0a8640 1 -- 172.21.15.112:0/1715241349 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36440b0910 con 0x7f36440aeae0 2024-03-29T10:52:34.174 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.187+0000 7f364b0a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:34.174 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.187+0000 7f364b0a8640 10 monclient(hunting): authenticate will time out at 909.492511s 2024-03-29T10:52:34.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.187+0000 7f36437fe640 1 -- 172.21.15.112:0/1715241349 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743453668 0 0) 0x7f36440b0910 con 0x7f36440aeae0 2024-03-29T10:52:34.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.187+0000 7f36437fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:34.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.187+0000 7f36437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:34.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.187+0000 7f36437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:34.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.187+0000 7f36437fe640 10 monclient(hunting): my global_id is 4271 2024-03-29T10:52:34.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.187+0000 7f36437fe640 1 -- 172.21.15.112:0/1715241349 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f362c003520 con 0x7f36440aeae0 2024-03-29T10:52:34.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.188+0000 7f36437fe640 1 -- 172.21.15.112:0/1715241349 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 105092408 0 0) 0x7f362c003520 con 0x7f36440aeae0 2024-03-29T10:52:34.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.188+0000 7f36437fe640 1 -- 172.21.15.112:0/1715241349 >> v1:172.21.15.118:6789/0 conn(0x7f36440b46b0 legacy=0x7f36440b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:34.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.188+0000 7f36437fe640 1 -- 172.21.15.112:0/1715241349 >> v1:172.21.15.112:6789/0 conn(0x7f364413fa10 legacy=0x7f364414d9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:34.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.188+0000 7f36437fe640 10 monclient: _finish_hunting 0 2024-03-29T10:52:34.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.188+0000 7f36437fe640 1 monclient: found mon.noname-b 2024-03-29T10:52:34.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.188+0000 7f36437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:34.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.188+0000 7f36437fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:52:34.176 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.188+0000 7f36437fe640 1 -- 172.21.15.112:0/1715241349 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36440af6d0 con 0x7f36440aeae0 2024-03-29T10:52:34.176 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.188+0000 7f36437fe640 10 monclient: _finish_auth 0 2024-03-29T10:52:34.176 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.188+0000 7f36437fe640 10 monclient: _check_auth_tickets 2024-03-29T10:52:34.176 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.188+0000 7f36437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:34.176 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.188+0000 7f36437fe640 1 -- 172.21.15.112:0/1715241349 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f3638001690 con 0x7f36440aeae0 2024-03-29T10:52:34.176 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.188+0000 7f36437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:34.176 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.188+0000 7f36437fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:34.176 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.188+0000 7f36437fe640 10 monclient: dump: 2024-03-29T10:52:34.176 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:34.176 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:34.176 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:34.176 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:34.176 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:34.176 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:34.177 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:34.177 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:34.177 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:34.177 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:34.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.188+0000 7f364b0a8640 5 monclient: authenticate success, global_id 4271 2024-03-29T10:52:34.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.188+0000 7f364b0a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:34.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.189+0000 7f36437fe640 1 -- 172.21.15.112:0/1715241349 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3638001920 con 0x7f36440aeae0 2024-03-29T10:52:34.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.189+0000 7f36437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:34.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.189+0000 7f36437fe640 1 -- 172.21.15.112:0/1715241349 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f3638001cb0 con 0x7f36440aeae0 2024-03-29T10:52:34.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.189+0000 7f36437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:34.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.189+0000 7f36437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:34.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.189+0000 7f36437fe640 10 monclient: dump: 2024-03-29T10:52:34.177 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:34.177 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:34.178 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:34.178 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:34.178 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:34.178 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:34.178 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:34.178 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:34.178 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:34.178 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:34.178 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.189+0000 7f364b0a8640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:34.178 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.189+0000 7f364b0a8640 4 set_mon_vals no callback set 2024-03-29T10:52:34.179 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.189+0000 7f364b0a8640 10 monclient: shutdown 2024-03-29T10:52:34.179 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.190+0000 7f364b0a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:34.179 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.190+0000 7f364b0a8640 1 -- 172.21.15.112:0/1715241349 >> v1:172.21.15.112:6790/0 conn(0x7f36440aeae0 legacy=0x7f364413f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:34.179 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.190+0000 7f364b0a8640 1 -- 172.21.15.112:0/1715241349 shutdown_connections 2024-03-29T10:52:34.179 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.190+0000 7f364b0a8640 1 -- 172.21.15.112:0/1715241349 >> 172.21.15.112:0/1715241349 conn(0x7f3644066a00 msgr2=0x7f3644066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:34.179 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.190+0000 7f364b0a8640 1 -- 172.21.15.112:0/1715241349 shutdown_connections 2024-03-29T10:52:34.179 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.190+0000 7f364b0a8640 1 -- 172.21.15.112:0/1715241349 wait complete. 2024-03-29T10:52:34.179 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.191+0000 7f364b0a8640 10 monclient: build_initial_monmap 2024-03-29T10:52:34.179 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.191+0000 7f364b0a8640 10 monclient: monmap: 2024-03-29T10:52:34.179 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:52:34.179 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:34.179 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:52:34.179 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:52:34.179 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:34.179 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:34.180 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:52:34.180 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:52:34.180 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:52:34.180 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:34.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.191+0000 7f364b0a8640 1 librados: starting msgr at 2024-03-29T10:52:34.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.191+0000 7f364b0a8640 1 librados: starting objecter 2024-03-29T10:52:34.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.191+0000 7f364b0a8640 1 Processor -- start 2024-03-29T10:52:34.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.191+0000 7f364b0a8640 1 -- start start 2024-03-29T10:52:34.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.191+0000 7f364b0a8640 1 librados: setting wanted keys 2024-03-29T10:52:34.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.191+0000 7f364b0a8640 1 librados: calling monclient init 2024-03-29T10:52:34.181 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.191+0000 7f364b0a8640 10 monclient: init 2024-03-29T10:52:34.185 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.192+0000 7f364b0a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:34.185 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.192+0000 7f364b0a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:34.185 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.192+0000 7f364b0a8640 10 monclient(hunting): picked mon.noname-c con 0x7f36440aeae0 addr v1:172.21.15.112:6790/0 2024-03-29T10:52:34.185 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.193+0000 7f364b0a8640 10 monclient(hunting): picked mon.noname-b con 0x7f36440b46b0 addr v1:172.21.15.118:6789/0 2024-03-29T10:52:34.185 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.193+0000 7f3643fff640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f36440b46b0 0x7f3644134ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:52448/0 (socket says 172.21.15.112:52448) 2024-03-29T10:52:34.185 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.193+0000 7f3648e44640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f36440aeae0 0x7f3644145c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:60264/0 (socket says 172.21.15.112:60264) 2024-03-29T10:52:34.185 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.194+0000 7f364b0a8640 10 monclient(hunting): picked mon.noname-a con 0x7f364413fa10 addr v1:172.21.15.112:6789/0 2024-03-29T10:52:34.185 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.194+0000 7f364b0a8640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3644141040 con 0x7f364413fa10 2024-03-29T10:52:34.185 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.194+0000 7f364b0a8640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36441456b0 con 0x7f36440b46b0 2024-03-29T10:52:34.185 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.194+0000 7f3643fff640 1 -- 172.21.15.112:0/3382101906 learned_addr learned my addr 172.21.15.112:0/3382101906 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:52:34.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.194+0000 7f364b0a8640 1 -- 172.21.15.112:0/3382101906 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3644146360 con 0x7f36440aeae0 2024-03-29T10:52:34.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.194+0000 7f364b0a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:34.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.194+0000 7f364b0a8640 10 monclient(hunting): authenticate will time out at 909.499526s 2024-03-29T10:52:34.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.194+0000 7f36417fa640 10 client.?.objecter ms_handle_connect 0x7f36440aeae0 2024-03-29T10:52:34.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.194+0000 7f36417fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:34.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.194+0000 7f36417fa640 1 -- 172.21.15.112:0/3382101906 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244273850 0 0) 0x7f3644146360 con 0x7f36440aeae0 2024-03-29T10:52:34.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.194+0000 7f36417fa640 10 monclient(hunting): _negotiate 2024-03-29T10:52:34.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.194+0000 7f36417fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:34.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.194+0000 7f36417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:34.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.194+0000 7f36417fa640 10 monclient(hunting): my global_id is 4274 2024-03-29T10:52:34.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.194+0000 7f36417fa640 1 -- 172.21.15.112:0/3382101906 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3630004eb0 con 0x7f36440aeae0 2024-03-29T10:52:34.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f36417fa640 1 -- 172.21.15.112:0/3382101906 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1438064385 0 0) 0x7f3630004eb0 con 0x7f36440aeae0 2024-03-29T10:52:34.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f36417fa640 1 -- 172.21.15.112:0/3382101906 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3644146360 con 0x7f36440aeae0 2024-03-29T10:52:34.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f36417fa640 1 -- 172.21.15.112:0/3382101906 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f363c001cb0 con 0x7f36440aeae0 2024-03-29T10:52:34.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f36417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:34.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f36417fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:34.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f36417fa640 10 monclient(hunting): dump: 2024-03-29T10:52:34.187 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:34.187 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:34.187 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:34.187 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:34.188 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:34.188 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:34.188 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:34.188 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:34.188 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:34.188 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:34.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f36417fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:34.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f36417fa640 1 -- 172.21.15.112:0/3382101906 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1783960287 0 0) 0x7f3644146360 con 0x7f36440aeae0 2024-03-29T10:52:34.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f36417fa640 1 -- 172.21.15.112:0/3382101906 >> v1:172.21.15.118:6789/0 conn(0x7f36440b46b0 legacy=0x7f3644134ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:34.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f36417fa640 1 -- 172.21.15.112:0/3382101906 >> v1:172.21.15.112:6789/0 conn(0x7f364413fa10 legacy=0x7f36441e2d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:52:34.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f36417fa640 10 monclient: _finish_hunting 0 2024-03-29T10:52:34.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f36417fa640 1 monclient: found mon.c 2024-03-29T10:52:34.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f36417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:34.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f36417fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:52:34.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f36417fa640 1 -- 172.21.15.112:0/3382101906 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3644145220 con 0x7f36440aeae0 2024-03-29T10:52:34.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f36417fa640 10 monclient: _finish_auth 0 2024-03-29T10:52:34.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f36417fa640 10 monclient: _check_auth_tickets 2024-03-29T10:52:34.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f36417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:34.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f364b0a8640 5 monclient: authenticate success, global_id 4274 2024-03-29T10:52:34.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f364b0a8640 10 monclient: _renew_subs 2024-03-29T10:52:34.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f364b0a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:52:34.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f364b0a8640 1 -- 172.21.15.112:0/3382101906 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364413fee0 con 0x7f36440aeae0 2024-03-29T10:52:34.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f364b0a8640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:34.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f364b0a8640 10 monclient: _renew_subs 2024-03-29T10:52:34.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f364b0a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:52:34.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.195+0000 7f364b0a8640 1 -- 172.21.15.112:0/3382101906 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36440b7350 con 0x7f36440aeae0 2024-03-29T10:52:34.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.196+0000 7f364b0a8640 1 librados: init done 2024-03-29T10:52:34.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.197+0000 7f36417fa640 1 -- 172.21.15.112:0/3382101906 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f363c001e10 con 0x7f36440aeae0 2024-03-29T10:52:34.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.197+0000 7f36417fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:34.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.197+0000 7f36417fa640 1 -- 172.21.15.112:0/3382101906 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f363c004e40 con 0x7f36440aeae0 2024-03-29T10:52:34.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.197+0000 7f36417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:34.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.197+0000 7f36417fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:34.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.197+0000 7f36417fa640 10 monclient: dump: 2024-03-29T10:52:34.190 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:34.190 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:34.190 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:34.190 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:34.190 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:34.190 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:34.190 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:34.190 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:34.190 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:34.190 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:34.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.197+0000 7f3642ffd640 4 set_mon_vals no callback set 2024-03-29T10:52:34.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.198+0000 7f364b0a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:34.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.198+0000 7f364b0a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:34.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.198+0000 7f364b0a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:52:34.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.198+0000 7f364b0a8640 1 -- 172.21.15.112:0/3382101906 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3624005080 con 0x7f36440aeae0 2024-03-29T10:52:34.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.198+0000 7f36417fa640 1 -- 172.21.15.112:0/3382101906 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f363c004fa0 con 0x7f36440aeae0 2024-03-29T10:52:34.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.199+0000 7f36417fa640 1 -- 172.21.15.112:0/3382101906 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f363c084e80 con 0x7f36440aeae0 2024-03-29T10:52:34.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.199+0000 7f36417fa640 10 client.4274.objecter ms_dispatch 0x7f3644000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:34.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.199+0000 7f36417fa640 3 client.4274.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:34.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.199+0000 7f36417fa640 3 client.4274.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:34.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.199+0000 7f36417fa640 20 client.4274.objecter dump_active .. 0 homeless 2024-03-29T10:52:34.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.199+0000 7f3643fff640 10 monclient: get_auth_request con 0x7f363006dbc0 auth_method 0 2024-03-29T10:52:34.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.200+0000 7f36417fa640 10 client.4274.objecter ms_handle_connect 0x7f363006dbc0 2024-03-29T10:52:34.192 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.205+0000 7f36417fa640 1 -- 172.21.15.112:0/3382101906 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f363c050760 con 0x7f36440aeae0 2024-03-29T10:52:34.192 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.205+0000 7f36417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:34.192 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.205+0000 7f36417fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:34.491 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.509+0000 7f364b0a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:34.492 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.509+0000 7f364b0a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:34.492 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.509+0000 7f364b0a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:52:34.492 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.509+0000 7f364b0a8640 1 -- 172.21.15.112:0/3382101906 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3624005330 con 0x7f36440aeae0 2024-03-29T10:52:34.492 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.510+0000 7f36417fa640 1 -- 172.21.15.112:0/3382101906 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f363c052690 con 0x7f36440aeae0 2024-03-29T10:52:34.492 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.510+0000 7f36417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:34.492 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.510+0000 7f36417fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:34.493 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:52:34.493 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:34.495 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.513+0000 7f364b0a8640 10 librados: watch_flush enter 2024-03-29T10:52:34.496 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.513+0000 7f364b0a8640 10 librados: watch_flush exit 2024-03-29T10:52:34.496 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.513+0000 7f364b0a8640 20 client.4274.objecter shutdown clearing up homeless session... 2024-03-29T10:52:34.496 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.513+0000 7f364b0a8640 10 client.4274.objecter successfully canceled tick 2024-03-29T10:52:34.496 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.513+0000 7f364b0a8640 1 -- 172.21.15.112:0/3382101906 >> v1:172.21.15.118:6816/3181073408 conn(0x7f363006dbc0 legacy=0x7f3630070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:34.496 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.513+0000 7f364b0a8640 10 monclient: shutdown 2024-03-29T10:52:34.496 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.513+0000 7f364b0a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:34.496 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.513+0000 7f364b0a8640 1 -- 172.21.15.112:0/3382101906 >> v1:172.21.15.112:6790/0 conn(0x7f36440aeae0 legacy=0x7f3644145c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:34.496 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.513+0000 7f364b0a8640 1 -- 172.21.15.112:0/3382101906 shutdown_connections 2024-03-29T10:52:34.496 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.513+0000 7f364b0a8640 1 -- 172.21.15.112:0/3382101906 >> 172.21.15.112:0/3382101906 conn(0x7f3644066a00 msgr2=0x7f36440b3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:34.496 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.513+0000 7f364b0a8640 1 -- 172.21.15.112:0/3382101906 shutdown_connections 2024-03-29T10:52:34.496 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.513+0000 7f364b0a8640 1 -- 172.21.15.112:0/3382101906 wait complete. 2024-03-29T10:52:34.496 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:34.513+0000 7f364b0a8640 1 librados: shutdown 2024-03-29T10:52:34.509 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:41.317 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:41.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.487+0000 7f5d502b3640 10 monclient: get_monmap_and_config 2024-03-29T10:52:41.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.487+0000 7f5d502b3640 10 monclient: build_initial_monmap 2024-03-29T10:52:41.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.487+0000 7f5d502b3640 10 monclient: monmap: 2024-03-29T10:52:41.470 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:52:41.470 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:41.471 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:52:41.488829+0000 2024-03-29T10:52:41.471 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:52:41.488829+0000 2024-03-29T10:52:41.471 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:41.471 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:41.471 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:52:41.471 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:52:41.471 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:52:41.471 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:41.471 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.489+0000 7f5d502b3640 1 Processor -- start 2024-03-29T10:52:41.472 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.489+0000 7f5d502b3640 1 -- start start 2024-03-29T10:52:41.472 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.489+0000 7f5d502b3640 10 monclient: init 2024-03-29T10:52:41.472 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.490+0000 7f5d502b3640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:41.472 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.490+0000 7f5d502b3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:41.472 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.490+0000 7f5d502b3640 10 monclient(hunting): picked mon.noname-c con 0x7f5d480a96c0 addr v1:172.21.15.118:6789/0 2024-03-29T10:52:41.472 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.490+0000 7f5d502b3640 10 monclient(hunting): picked mon.noname-a con 0x7f5d480a9aa0 addr v1:172.21.15.112:6789/0 2024-03-29T10:52:41.472 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.490+0000 7f5d502b3640 10 monclient(hunting): picked mon.noname-b con 0x7f5d480ad730 addr v1:172.21.15.112:6790/0 2024-03-29T10:52:41.472 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.490+0000 7f5d502b3640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d4813fee0 con 0x7f5d480a9aa0 2024-03-29T10:52:41.472 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.490+0000 7f5d502b3640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d480aab80 con 0x7f5d480a96c0 2024-03-29T10:52:41.472 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.490+0000 7f5d502b3640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d48145e80 con 0x7f5d480ad730 2024-03-29T10:52:41.473 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.490+0000 7f5d502b3640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:41.473 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.490+0000 7f5d502b3640 10 monclient(hunting): authenticate will time out at 916.795156s 2024-03-29T10:52:41.473 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.491+0000 7f5d4e04f640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f5d480a96c0 0x7f5d4813f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:43660/0 (socket says 172.21.15.112:43660) 2024-03-29T10:52:41.473 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.491+0000 7f5d4e04f640 1 -- 172.21.15.112:0/1280925616 learned_addr learned my addr 172.21.15.112:0/1280925616 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:52:41.473 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.491+0000 7f5d4d04d640 1 -- 172.21.15.112:0/1280925616 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439901224 0 0) 0x7f5d480aab80 con 0x7f5d480a96c0 2024-03-29T10:52:41.473 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.491+0000 7f5d4d04d640 10 monclient(hunting): _negotiate 2024-03-29T10:52:41.474 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.491+0000 7f5d4d04d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:41.474 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.491+0000 7f5d4d04d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:41.474 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.491+0000 7f5d4d04d640 10 monclient(hunting): my global_id is 4288 2024-03-29T10:52:41.474 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.491+0000 7f5d4d04d640 1 -- 172.21.15.112:0/1280925616 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d34003520 con 0x7f5d480a96c0 2024-03-29T10:52:41.474 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.492+0000 7f5d4d04d640 1 -- 172.21.15.112:0/1280925616 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1072919206 0 0) 0x7f5d34003520 con 0x7f5d480a96c0 2024-03-29T10:52:41.474 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.492+0000 7f5d4d04d640 1 -- 172.21.15.112:0/1280925616 >> v1:172.21.15.112:6790/0 conn(0x7f5d480ad730 legacy=0x7f5d4814dad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:41.474 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.492+0000 7f5d4d04d640 1 -- 172.21.15.112:0/1280925616 >> v1:172.21.15.112:6789/0 conn(0x7f5d480a9aa0 legacy=0x7f5d48145770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:41.476 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.494+0000 7f5d4d04d640 10 monclient: _finish_hunting 0 2024-03-29T10:52:41.476 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.494+0000 7f5d4d04d640 1 monclient: found mon.noname-c 2024-03-29T10:52:41.476 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.494+0000 7f5d4d04d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:41.477 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.494+0000 7f5d4d04d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T10:52:41.477 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.494+0000 7f5d4d04d640 1 -- 172.21.15.112:0/1280925616 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d480b4440 con 0x7f5d480a96c0 2024-03-29T10:52:41.477 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.494+0000 7f5d4d04d640 10 monclient: _finish_auth 0 2024-03-29T10:52:41.477 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.494+0000 7f5d4d04d640 10 monclient: _check_auth_tickets 2024-03-29T10:52:41.477 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.494+0000 7f5d4d04d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:41.477 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.494+0000 7f5d502b3640 5 monclient: authenticate success, global_id 4288 2024-03-29T10:52:41.477 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.494+0000 7f5d502b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:41.477 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.494+0000 7f5d4d04d640 1 -- 172.21.15.112:0/1280925616 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f5d38001690 con 0x7f5d480a96c0 2024-03-29T10:52:41.477 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.494+0000 7f5d4d04d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:41.477 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.494+0000 7f5d4d04d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:41.477 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.494+0000 7f5d4d04d640 10 monclient: dump: 2024-03-29T10:52:41.477 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:41.477 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:41.477 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:41.478 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:41.478 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:41.478 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:41.478 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:41.478 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:41.478 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:41.478 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:41.478 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.494+0000 7f5d502b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:41.478 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.495+0000 7f5d4d04d640 1 -- 172.21.15.112:0/1280925616 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d38001920 con 0x7f5d480a96c0 2024-03-29T10:52:41.478 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.495+0000 7f5d4d04d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:41.479 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.495+0000 7f5d502b3640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:41.479 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.495+0000 7f5d502b3640 4 set_mon_vals no callback set 2024-03-29T10:52:41.479 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.495+0000 7f5d502b3640 10 monclient: shutdown 2024-03-29T10:52:41.479 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.495+0000 7f5d502b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:41.479 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.495+0000 7f5d502b3640 1 -- 172.21.15.112:0/1280925616 >> v1:172.21.15.118:6789/0 conn(0x7f5d480a96c0 legacy=0x7f5d4813f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:41.479 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.496+0000 7f5d502b3640 1 -- 172.21.15.112:0/1280925616 shutdown_connections 2024-03-29T10:52:41.479 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.496+0000 7f5d502b3640 1 -- 172.21.15.112:0/1280925616 >> 172.21.15.112:0/1280925616 conn(0x7f5d480b35e0 msgr2=0x7f5d480b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:41.479 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.496+0000 7f5d502b3640 1 -- 172.21.15.112:0/1280925616 shutdown_connections 2024-03-29T10:52:41.479 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.496+0000 7f5d502b3640 1 -- 172.21.15.112:0/1280925616 wait complete. 2024-03-29T10:52:41.481 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.497+0000 7f5d502b3640 10 monclient: build_initial_monmap 2024-03-29T10:52:41.481 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.497+0000 7f5d502b3640 10 monclient: monmap: 2024-03-29T10:52:41.481 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:52:41.481 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:41.481 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:52:41.481 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:52:41.481 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:41.481 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:41.481 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:52:41.482 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:52:41.482 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:52:41.482 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:41.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.498+0000 7f5d502b3640 1 librados: starting msgr at 2024-03-29T10:52:41.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.498+0000 7f5d502b3640 1 librados: starting objecter 2024-03-29T10:52:41.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.498+0000 7f5d502b3640 1 Processor -- start 2024-03-29T10:52:41.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.498+0000 7f5d502b3640 1 -- start start 2024-03-29T10:52:41.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.498+0000 7f5d502b3640 1 librados: setting wanted keys 2024-03-29T10:52:41.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.498+0000 7f5d502b3640 1 librados: calling monclient init 2024-03-29T10:52:41.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.498+0000 7f5d502b3640 10 monclient: init 2024-03-29T10:52:41.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.499+0000 7f5d502b3640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:41.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.499+0000 7f5d502b3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:41.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.499+0000 7f5d502b3640 10 monclient(hunting): picked mon.noname-b con 0x7f5d480a96c0 addr v1:172.21.15.118:6789/0 2024-03-29T10:52:41.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.499+0000 7f5d502b3640 10 monclient(hunting): picked mon.noname-a con 0x7f5d480ad730 addr v1:172.21.15.112:6789/0 2024-03-29T10:52:41.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.500+0000 7f5d502b3640 10 monclient(hunting): picked mon.noname-c con 0x7f5d4813c720 addr v1:172.21.15.112:6790/0 2024-03-29T10:52:41.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.500+0000 7f5d502b3640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d481e7530 con 0x7f5d480ad730 2024-03-29T10:52:41.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.500+0000 7f5d502b3640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d4813e550 con 0x7f5d480a96c0 2024-03-29T10:52:41.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.500+0000 7f5d502b3640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d4813e830 con 0x7f5d4813c720 2024-03-29T10:52:41.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.500+0000 7f5d502b3640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:41.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.500+0000 7f5d502b3640 10 monclient(hunting): authenticate will time out at 916.804401s 2024-03-29T10:52:41.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.500+0000 7f5d4e04f640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f5d480a96c0 0x7f5d48139270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:43930/0 (socket says 172.21.15.112:43930) 2024-03-29T10:52:41.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.500+0000 7f5d4e04f640 1 -- 172.21.15.112:0/1065826537 learned_addr learned my addr 172.21.15.112:0/1065826537 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:52:41.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.500+0000 7f5d3effd640 10 client.?.objecter ms_handle_connect 0x7f5d480a96c0 2024-03-29T10:52:41.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.500+0000 7f5d3effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:41.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.500+0000 7f5d3effd640 1 -- 172.21.15.112:0/1065826537 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285755807 0 0) 0x7f5d4813e550 con 0x7f5d480a96c0 2024-03-29T10:52:41.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.501+0000 7f5d3effd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:41.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.501+0000 7f5d3effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:41.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.501+0000 7f5d3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:41.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.501+0000 7f5d3effd640 10 monclient(hunting): my global_id is 4291 2024-03-29T10:52:41.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.501+0000 7f5d3effd640 1 -- 172.21.15.112:0/1065826537 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d2c004eb0 con 0x7f5d480a96c0 2024-03-29T10:52:41.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.501+0000 7f5d3effd640 1 -- 172.21.15.112:0/1065826537 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2843620894 0 0) 0x7f5d2c004eb0 con 0x7f5d480a96c0 2024-03-29T10:52:41.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.501+0000 7f5d3effd640 1 -- 172.21.15.112:0/1065826537 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d4813e550 con 0x7f5d480a96c0 2024-03-29T10:52:41.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.501+0000 7f5d3effd640 1 -- 172.21.15.112:0/1065826537 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f5d380049c0 con 0x7f5d480a96c0 2024-03-29T10:52:41.486 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.501+0000 7f5d3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:41.486 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.501+0000 7f5d3effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:41.486 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.501+0000 7f5d3effd640 10 monclient(hunting): dump: 2024-03-29T10:52:41.486 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:41.486 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:41.486 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:41.486 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:41.486 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:41.486 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:41.486 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:41.486 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:41.486 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:41.486 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:41.486 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.501+0000 7f5d3effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:41.486 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.501+0000 7f5d3effd640 1 -- 172.21.15.112:0/1065826537 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3717342675 0 0) 0x7f5d4813e550 con 0x7f5d480a96c0 2024-03-29T10:52:41.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.501+0000 7f5d3effd640 1 -- 172.21.15.112:0/1065826537 >> v1:172.21.15.112:6790/0 conn(0x7f5d4813c720 legacy=0x7f5d481e6e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:52:41.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.506+0000 7f5d3effd640 1 -- 172.21.15.112:0/1065826537 >> v1:172.21.15.112:6789/0 conn(0x7f5d480ad730 legacy=0x7f5d481e6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:41.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.506+0000 7f5d3effd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:41.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.506+0000 7f5d3effd640 1 monclient: found mon.b 2024-03-29T10:52:41.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.506+0000 7f5d3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:41.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.506+0000 7f5d3effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:52:41.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.506+0000 7f5d3effd640 1 -- 172.21.15.112:0/1065826537 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d4806e8d0 con 0x7f5d480a96c0 2024-03-29T10:52:41.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.506+0000 7f5d3effd640 10 monclient: _finish_auth 0 2024-03-29T10:52:41.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.506+0000 7f5d3effd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:41.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.506+0000 7f5d3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:41.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.506+0000 7f5d502b3640 5 monclient: authenticate success, global_id 4291 2024-03-29T10:52:41.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.506+0000 7f5d502b3640 10 monclient: _renew_subs 2024-03-29T10:52:41.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.506+0000 7f5d502b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:52:41.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.506+0000 7f5d502b3640 1 -- 172.21.15.112:0/1065826537 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d4813d330 con 0x7f5d480a96c0 2024-03-29T10:52:41.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.506+0000 7f5d502b3640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:41.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.506+0000 7f5d502b3640 10 monclient: _renew_subs 2024-03-29T10:52:41.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.506+0000 7f5d502b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:52:41.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.506+0000 7f5d502b3640 1 -- 172.21.15.112:0/1065826537 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d48139980 con 0x7f5d480a96c0 2024-03-29T10:52:41.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.506+0000 7f5d502b3640 1 librados: init done 2024-03-29T10:52:41.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.506+0000 7f5d3effd640 1 -- 172.21.15.112:0/1065826537 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d38001b10 con 0x7f5d480a96c0 2024-03-29T10:52:41.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.506+0000 7f5d3effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:41.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.506+0000 7f5d3effd640 1 -- 172.21.15.112:0/1065826537 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f5d380050b0 con 0x7f5d480a96c0 2024-03-29T10:52:41.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.506+0000 7f5d4c84c640 4 set_mon_vals no callback set 2024-03-29T10:52:41.490 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.507+0000 7f5d3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:41.490 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.507+0000 7f5d3effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:41.490 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.507+0000 7f5d3effd640 10 monclient: dump: 2024-03-29T10:52:41.490 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:41.490 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:41.490 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:41.490 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:41.491 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:41.491 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:41.491 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:41.491 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:41.491 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:41.491 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:41.491 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.509+0000 7f5d3effd640 1 -- 172.21.15.112:0/1065826537 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f5d38005210 con 0x7f5d480a96c0 2024-03-29T10:52:41.492 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.510+0000 7f5d3effd640 1 -- 172.21.15.112:0/1065826537 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f5d38084e80 con 0x7f5d480a96c0 2024-03-29T10:52:41.492 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.510+0000 7f5d3effd640 10 client.4291.objecter ms_dispatch 0x7f5d48000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:41.492 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.510+0000 7f5d3effd640 3 client.4291.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:41.493 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.510+0000 7f5d3effd640 3 client.4291.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:41.493 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.510+0000 7f5d3effd640 20 client.4291.objecter dump_active .. 0 homeless 2024-03-29T10:52:41.493 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.510+0000 7f5d502b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:41.493 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.510+0000 7f5d502b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:41.493 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.510+0000 7f5d502b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:52:41.493 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.510+0000 7f5d502b3640 1 -- 172.21.15.112:0/1065826537 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d30005080 con 0x7f5d480a96c0 2024-03-29T10:52:41.493 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.511+0000 7f5d4d84e640 10 monclient: get_auth_request con 0x7f5d2c06db60 auth_method 0 2024-03-29T10:52:41.493 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.511+0000 7f5d3effd640 10 client.4291.objecter ms_handle_connect 0x7f5d2c06db60 2024-03-29T10:52:41.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.521+0000 7f5d3effd640 1 -- 172.21.15.112:0/1065826537 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5d38050760 con 0x7f5d480a96c0 2024-03-29T10:52:41.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.521+0000 7f5d3effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:41.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.521+0000 7f5d3effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:41.831 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.850+0000 7f5d502b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:41.831 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.850+0000 7f5d502b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:41.832 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.850+0000 7f5d502b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:52:41.832 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.850+0000 7f5d502b3640 1 -- 172.21.15.112:0/1065826537 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d30005600 con 0x7f5d480a96c0 2024-03-29T10:52:41.832 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.851+0000 7f5d3effd640 1 -- 172.21.15.112:0/1065826537 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f5d38052690 con 0x7f5d480a96c0 2024-03-29T10:52:41.832 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.851+0000 7f5d3effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:41.832 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.851+0000 7f5d3effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:41.833 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:52:41.833 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:41.836 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.855+0000 7f5d502b3640 10 librados: watch_flush enter 2024-03-29T10:52:41.836 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.855+0000 7f5d502b3640 10 librados: watch_flush exit 2024-03-29T10:52:41.836 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.855+0000 7f5d502b3640 20 client.4291.objecter shutdown clearing up homeless session... 2024-03-29T10:52:41.836 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.855+0000 7f5d502b3640 10 client.4291.objecter successfully canceled tick 2024-03-29T10:52:41.837 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.856+0000 7f5d502b3640 1 -- 172.21.15.112:0/1065826537 >> v1:172.21.15.118:6816/3181073408 conn(0x7f5d2c06db60 legacy=0x7f5d2c070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:41.837 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.856+0000 7f5d502b3640 10 monclient: shutdown 2024-03-29T10:52:41.837 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.856+0000 7f5d502b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:41.838 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.856+0000 7f5d502b3640 1 -- 172.21.15.112:0/1065826537 >> v1:172.21.15.118:6789/0 conn(0x7f5d480a96c0 legacy=0x7f5d48139270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:41.838 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.856+0000 7f5d502b3640 1 -- 172.21.15.112:0/1065826537 shutdown_connections 2024-03-29T10:52:41.838 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.856+0000 7f5d502b3640 1 -- 172.21.15.112:0/1065826537 >> 172.21.15.112:0/1065826537 conn(0x7f5d480b35e0 msgr2=0x7f5d480b00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:41.838 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.856+0000 7f5d502b3640 1 -- 172.21.15.112:0/1065826537 shutdown_connections 2024-03-29T10:52:41.838 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.856+0000 7f5d502b3640 1 -- 172.21.15.112:0/1065826537 wait complete. 2024-03-29T10:52:41.838 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:41.856+0000 7f5d502b3640 1 librados: shutdown 2024-03-29T10:52:41.854 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:48.661 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:48.817 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.835+0000 7fe033fff640 10 monclient: get_monmap_and_config 2024-03-29T10:52:48.817 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.835+0000 7fe033fff640 10 monclient: build_initial_monmap 2024-03-29T10:52:48.817 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.835+0000 7fe033fff640 10 monclient: monmap: 2024-03-29T10:52:48.817 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:52:48.817 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:48.817 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:52:48.836572+0000 2024-03-29T10:52:48.817 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:52:48.836572+0000 2024-03-29T10:52:48.817 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:48.817 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:48.817 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:52:48.817 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:52:48.818 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:52:48.818 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:48.819 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.836+0000 7fe033fff640 1 Processor -- start 2024-03-29T10:52:48.819 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.836+0000 7fe033fff640 1 -- start start 2024-03-29T10:52:48.819 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.836+0000 7fe033fff640 10 monclient: init 2024-03-29T10:52:48.819 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.837+0000 7fe033fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:48.819 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.837+0000 7fe033fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:52:48.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.839+0000 7fe033fff640 10 monclient(hunting): picked mon.noname-c con 0x7fe02c006e30 addr v1:172.21.15.118:6789/0 2024-03-29T10:52:48.823 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.841+0000 7fe033fff640 10 monclient(hunting): picked mon.noname-b con 0x7fe02c035e00 addr v1:172.21.15.112:6790/0 2024-03-29T10:52:48.823 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.841+0000 7fe033fff640 10 monclient(hunting): picked mon.noname-a con 0x7fe02c0c9670 addr v1:172.21.15.112:6789/0 2024-03-29T10:52:48.823 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.841+0000 7fe033fff640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02c035970 con 0x7fe02c0c9670 2024-03-29T10:52:48.823 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.841+0000 7fe033fff640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02c036280 con 0x7fe02c006e30 2024-03-29T10:52:48.823 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.841+0000 7fe032ffd640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fe02c006e30 0x7fe02c0cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:42509/0 (socket says 172.21.15.112:42509) 2024-03-29T10:52:48.823 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.841+0000 7fe032ffd640 1 -- 172.21.15.112:0/722092820 learned_addr learned my addr 172.21.15.112:0/722092820 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:52:48.823 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.841+0000 7fe033fff640 1 -- 172.21.15.112:0/722092820 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02c035640 con 0x7fe02c035e00 2024-03-29T10:52:48.823 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.841+0000 7fe033fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:48.823 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.841+0000 7fe033fff640 10 monclient(hunting): authenticate will time out at 924.145916s 2024-03-29T10:52:48.826 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 1 -- 172.21.15.112:0/722092820 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3520845004 0 0) 0x7fe02c035640 con 0x7fe02c035e00 2024-03-29T10:52:48.826 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:48.827 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:48.827 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:48.827 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 10 monclient(hunting): my global_id is 4277 2024-03-29T10:52:48.827 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 1 -- 172.21.15.112:0/722092820 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe01c003520 con 0x7fe02c035e00 2024-03-29T10:52:48.827 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 1 -- 172.21.15.112:0/722092820 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297198796 0 0) 0x7fe02c035970 con 0x7fe02c0c9670 2024-03-29T10:52:48.828 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:48.828 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:48.828 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:48.828 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 10 monclient(hunting): my global_id is 4278 2024-03-29T10:52:48.828 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 1 -- 172.21.15.112:0/722092820 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe02c035640 con 0x7fe02c0c9670 2024-03-29T10:52:48.828 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 1 -- 172.21.15.112:0/722092820 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3587607794 0 0) 0x7fe01c003520 con 0x7fe02c035e00 2024-03-29T10:52:48.828 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 1 -- 172.21.15.112:0/722092820 >> v1:172.21.15.118:6789/0 conn(0x7fe02c006e30 legacy=0x7fe02c0cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:52:48.829 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 1 -- 172.21.15.112:0/722092820 >> v1:172.21.15.112:6789/0 conn(0x7fe02c0c9670 legacy=0x7fe02c0d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:48.829 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:52:48.829 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 1 monclient: found mon.noname-b 2024-03-29T10:52:48.829 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:48.829 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:52:48.829 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 1 -- 172.21.15.112:0/722092820 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02c034e20 con 0x7fe02c035e00 2024-03-29T10:52:48.829 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 10 monclient: _finish_auth 0 2024-03-29T10:52:48.829 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:52:48.829 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:48.829 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 1 -- 172.21.15.112:0/722092820 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe028001690 con 0x7fe02c035e00 2024-03-29T10:52:48.829 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:48.829 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:48.829 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 10 monclient: dump: 2024-03-29T10:52:48.830 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:48.830 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:48.830 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:48.830 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:48.830 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:48.830 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:48.830 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:48.830 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:48.830 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:48.830 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:48.831 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 1 -- 172.21.15.112:0/722092820 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0280017f0 con 0x7fe02c035e00 2024-03-29T10:52:48.831 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:48.831 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 1 -- 172.21.15.112:0/722092820 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe028001c00 con 0x7fe02c035e00 2024-03-29T10:52:48.831 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:48.831 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:48.831 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.845+0000 7fe031ffb640 10 monclient: dump: 2024-03-29T10:52:48.831 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:48.831 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:48.831 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:48.831 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:48.831 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:48.831 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:48.831 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:48.831 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:48.831 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:48.832 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:48.832 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.846+0000 7fe033fff640 5 monclient: authenticate success, global_id 4277 2024-03-29T10:52:48.832 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.846+0000 7fe033fff640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:48.832 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.846+0000 7fe033fff640 4 set_mon_vals no callback set 2024-03-29T10:52:48.833 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.847+0000 7fe033fff640 10 monclient: shutdown 2024-03-29T10:52:48.833 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.847+0000 7fe033fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:48.833 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.847+0000 7fe033fff640 1 -- 172.21.15.112:0/722092820 >> v1:172.21.15.112:6790/0 conn(0x7fe02c035e00 legacy=0x7fe02c0c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:48.833 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.851+0000 7fe033fff640 1 -- 172.21.15.112:0/722092820 shutdown_connections 2024-03-29T10:52:48.833 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.851+0000 7fe033fff640 1 -- 172.21.15.112:0/722092820 >> 172.21.15.112:0/722092820 conn(0x7fe02c005c00 msgr2=0x7fe02c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:48.833 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.852+0000 7fe033fff640 1 -- 172.21.15.112:0/722092820 shutdown_connections 2024-03-29T10:52:48.835 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.852+0000 7fe033fff640 1 -- 172.21.15.112:0/722092820 wait complete. 2024-03-29T10:52:48.835 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.853+0000 7fe033fff640 10 monclient: build_initial_monmap 2024-03-29T10:52:48.835 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.853+0000 7fe033fff640 10 monclient: monmap: 2024-03-29T10:52:48.835 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:52:48.835 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:48.835 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:52:48.835 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:52:48.835 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:48.835 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:48.835 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:52:48.836 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:52:48.836 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:52:48.836 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:48.837 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.854+0000 7fe033fff640 1 librados: starting msgr at 2024-03-29T10:52:48.837 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.854+0000 7fe033fff640 1 librados: starting objecter 2024-03-29T10:52:48.837 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.854+0000 7fe033fff640 1 Processor -- start 2024-03-29T10:52:48.837 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.855+0000 7fe033fff640 1 -- start start 2024-03-29T10:52:48.837 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.855+0000 7fe033fff640 1 librados: setting wanted keys 2024-03-29T10:52:48.837 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.855+0000 7fe033fff640 1 librados: calling monclient init 2024-03-29T10:52:48.837 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.855+0000 7fe033fff640 10 monclient: init 2024-03-29T10:52:48.840 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.855+0000 7fe033fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:48.840 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.855+0000 7fe033fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:48.840 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.855+0000 7fe033fff640 10 monclient(hunting): picked mon.noname-b con 0x7fe02c006e30 addr v1:172.21.15.118:6789/0 2024-03-29T10:52:48.840 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.856+0000 7fe033fff640 10 monclient(hunting): picked mon.noname-a con 0x7fe02c035e00 addr v1:172.21.15.112:6789/0 2024-03-29T10:52:48.841 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.856+0000 7fe033fff640 10 monclient(hunting): picked mon.noname-c con 0x7fe02c0c9670 addr v1:172.21.15.112:6790/0 2024-03-29T10:52:48.841 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.856+0000 7fe033fff640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02c1640e0 con 0x7fe02c035e00 2024-03-29T10:52:48.841 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.856+0000 7fe033fff640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02c0320c0 con 0x7fe02c006e30 2024-03-29T10:52:48.841 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.856+0000 7fe033fff640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02c02fe60 con 0x7fe02c0c9670 2024-03-29T10:52:48.841 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.856+0000 7fe033fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:48.841 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.856+0000 7fe033fff640 10 monclient(hunting): authenticate will time out at 924.160233s 2024-03-29T10:52:48.841 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.859+0000 7fe0327fc640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fe02c035e00 0x7fe02c0c29d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:40937/0 (socket says 172.21.15.112:40937) 2024-03-29T10:52:48.841 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.859+0000 7fe0327fc640 1 -- 172.21.15.112:0/333321269 learned_addr learned my addr 172.21.15.112:0/333321269 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:52:48.846 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.859+0000 7fe01bfff640 10 client.?.objecter ms_handle_connect 0x7fe02c0c9670 2024-03-29T10:52:48.846 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.859+0000 7fe01bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:48.846 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.860+0000 7fe01bfff640 1 -- 172.21.15.112:0/333321269 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 167758630 0 0) 0x7fe02c02fe60 con 0x7fe02c0c9670 2024-03-29T10:52:48.846 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.860+0000 7fe01bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:48.846 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.860+0000 7fe01bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:48.847 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.860+0000 7fe01bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:48.847 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.860+0000 7fe01bfff640 10 monclient(hunting): my global_id is 4280 2024-03-29T10:52:48.847 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.860+0000 7fe01bfff640 1 -- 172.21.15.112:0/333321269 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe020004eb0 con 0x7fe02c0c9670 2024-03-29T10:52:48.847 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.860+0000 7fe01bfff640 1 -- 172.21.15.112:0/333321269 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2186185257 0 0) 0x7fe020004eb0 con 0x7fe02c0c9670 2024-03-29T10:52:48.847 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.860+0000 7fe01bfff640 1 -- 172.21.15.112:0/333321269 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe020005300 con 0x7fe02c0c9670 2024-03-29T10:52:48.847 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.860+0000 7fe01bfff640 1 -- 172.21.15.112:0/333321269 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe0240017d0 con 0x7fe02c0c9670 2024-03-29T10:52:48.847 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.860+0000 7fe01bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:48.847 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.860+0000 7fe01bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:48.847 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.860+0000 7fe01bfff640 10 monclient(hunting): dump: 2024-03-29T10:52:48.847 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:48.847 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:48.847 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:48.848 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:48.848 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:48.848 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:48.848 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:48.848 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:48.848 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:48.848 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:48.848 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.860+0000 7fe01bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:48.848 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.860+0000 7fe01bfff640 1 -- 172.21.15.112:0/333321269 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3828463488 0 0) 0x7fe020005300 con 0x7fe02c0c9670 2024-03-29T10:52:48.848 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.860+0000 7fe01bfff640 1 -- 172.21.15.112:0/333321269 >> v1:172.21.15.118:6789/0 conn(0x7fe02c006e30 legacy=0x7fe02c164570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:52:48.848 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.862+0000 7fe01bfff640 1 -- 172.21.15.112:0/333321269 >> v1:172.21.15.112:6789/0 conn(0x7fe02c035e00 legacy=0x7fe02c0c29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:48.848 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.862+0000 7fe01bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:52:48.848 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.862+0000 7fe01bfff640 1 monclient: found mon.c 2024-03-29T10:52:48.849 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.862+0000 7fe01bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:48.849 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.862+0000 7fe01bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:52:48.849 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.862+0000 7fe01bfff640 1 -- 172.21.15.112:0/333321269 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02c0c3690 con 0x7fe02c0c9670 2024-03-29T10:52:48.849 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.862+0000 7fe01bfff640 10 monclient: _finish_auth 0 2024-03-29T10:52:48.849 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.862+0000 7fe01bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:52:48.849 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.862+0000 7fe01bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:48.849 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.862+0000 7fe033fff640 5 monclient: authenticate success, global_id 4280 2024-03-29T10:52:48.849 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.862+0000 7fe033fff640 10 monclient: _renew_subs 2024-03-29T10:52:48.849 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.862+0000 7fe033fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:52:48.849 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.862+0000 7fe033fff640 1 -- 172.21.15.112:0/333321269 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe02c02e5c0 con 0x7fe02c0c9670 2024-03-29T10:52:48.849 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.862+0000 7fe033fff640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:48.849 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.862+0000 7fe033fff640 10 monclient: _renew_subs 2024-03-29T10:52:48.849 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.862+0000 7fe033fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:52:48.850 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.862+0000 7fe033fff640 1 -- 172.21.15.112:0/333321269 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe02c164c80 con 0x7fe02c0c9670 2024-03-29T10:52:48.850 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.862+0000 7fe033fff640 1 librados: init done 2024-03-29T10:52:48.850 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.863+0000 7fe01bfff640 1 -- 172.21.15.112:0/333321269 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0240035f0 con 0x7fe02c0c9670 2024-03-29T10:52:48.850 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.863+0000 7fe01bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:48.850 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.863+0000 7fe01bfff640 1 -- 172.21.15.112:0/333321269 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe0240017d0 con 0x7fe02c0c9670 2024-03-29T10:52:48.850 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.863+0000 7fe01bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:48.850 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.863+0000 7fe01bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:48.850 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.863+0000 7fe0317fa640 4 set_mon_vals no callback set 2024-03-29T10:52:48.850 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.863+0000 7fe01bfff640 10 monclient: dump: 2024-03-29T10:52:48.850 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:48.850 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:48.850 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:48.850 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:48.851 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:48.851 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:48.851 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:48.851 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:48.851 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:48.851 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:48.851 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.864+0000 7fe033fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:48.851 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.864+0000 7fe033fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:48.851 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.864+0000 7fe033fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:52:48.851 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.864+0000 7fe033fff640 1 -- 172.21.15.112:0/333321269 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe014005080 con 0x7fe02c0c9670 2024-03-29T10:52:48.851 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.865+0000 7fe01bfff640 1 -- 172.21.15.112:0/333321269 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fe0240195d0 con 0x7fe02c0c9670 2024-03-29T10:52:48.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.865+0000 7fe01bfff640 1 -- 172.21.15.112:0/333321269 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7fe024083520 con 0x7fe02c0c9670 2024-03-29T10:52:48.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.865+0000 7fe01bfff640 10 client.4280.objecter ms_dispatch 0x7fe034000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:48.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.865+0000 7fe01bfff640 3 client.4280.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:48.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.865+0000 7fe01bfff640 3 client.4280.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:48.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.865+0000 7fe032ffd640 10 monclient: get_auth_request con 0x7fe02006dfb0 auth_method 0 2024-03-29T10:52:48.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.865+0000 7fe01bfff640 20 client.4280.objecter dump_active .. 0 homeless 2024-03-29T10:52:48.853 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.871+0000 7fe01bfff640 10 client.4280.objecter ms_handle_connect 0x7fe02006dfb0 2024-03-29T10:52:48.857 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.875+0000 7fe01bfff640 1 -- 172.21.15.112:0/333321269 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe02404ee00 con 0x7fe02c0c9670 2024-03-29T10:52:48.857 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.875+0000 7fe01bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:48.857 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:48.875+0000 7fe01bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:49.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:49.201+0000 7fe033fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:49.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:49.201+0000 7fe033fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:49.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:49.201+0000 7fe033fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:52:49.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:49.201+0000 7fe033fff640 1 -- 172.21.15.112:0/333321269 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe014005330 con 0x7fe02c0c9670 2024-03-29T10:52:49.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:49.202+0000 7fe01bfff640 1 -- 172.21.15.112:0/333321269 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7fe024050d30 con 0x7fe02c0c9670 2024-03-29T10:52:49.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:49.202+0000 7fe01bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:49.184 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:49.202+0000 7fe01bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:49.185 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:52:49.185 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:49.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:49.207+0000 7fe033fff640 10 librados: watch_flush enter 2024-03-29T10:52:49.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:49.207+0000 7fe033fff640 10 librados: watch_flush exit 2024-03-29T10:52:49.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:49.207+0000 7fe033fff640 20 client.4280.objecter shutdown clearing up homeless session... 2024-03-29T10:52:49.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:49.207+0000 7fe033fff640 10 client.4280.objecter successfully canceled tick 2024-03-29T10:52:49.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:49.207+0000 7fe033fff640 1 -- 172.21.15.112:0/333321269 >> v1:172.21.15.118:6816/3181073408 conn(0x7fe02006dfb0 legacy=0x7fe020070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:49.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:49.207+0000 7fe033fff640 10 monclient: shutdown 2024-03-29T10:52:49.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:49.207+0000 7fe033fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:49.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:49.207+0000 7fe033fff640 1 -- 172.21.15.112:0/333321269 >> v1:172.21.15.112:6790/0 conn(0x7fe02c0c9670 legacy=0x7fe02c16c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:49.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:49.207+0000 7fe033fff640 1 -- 172.21.15.112:0/333321269 shutdown_connections 2024-03-29T10:52:49.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:49.207+0000 7fe033fff640 1 -- 172.21.15.112:0/333321269 >> 172.21.15.112:0/333321269 conn(0x7fe02c005c00 msgr2=0x7fe02c0c8480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:49.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:49.208+0000 7fe033fff640 1 -- 172.21.15.112:0/333321269 shutdown_connections 2024-03-29T10:52:49.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:49.208+0000 7fe033fff640 1 -- 172.21.15.112:0/333321269 wait complete. 2024-03-29T10:52:49.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:49.208+0000 7fe033fff640 1 librados: shutdown 2024-03-29T10:52:49.209 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:56.016 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:56.174 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.193+0000 7feff184a640 10 monclient: get_monmap_and_config 2024-03-29T10:52:56.174 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.193+0000 7feff184a640 10 monclient: build_initial_monmap 2024-03-29T10:52:56.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.193+0000 7feff184a640 10 monclient: monmap: 2024-03-29T10:52:56.175 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:52:56.175 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:56.175 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:52:56.194274+0000 2024-03-29T10:52:56.175 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:52:56.194274+0000 2024-03-29T10:52:56.175 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:56.175 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:56.175 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:52:56.175 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:52:56.175 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:52:56.175 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:56.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.194+0000 7feff184a640 1 Processor -- start 2024-03-29T10:52:56.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.194+0000 7feff184a640 1 -- start start 2024-03-29T10:52:56.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.194+0000 7feff184a640 10 monclient: init 2024-03-29T10:52:56.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.195+0000 7feff184a640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:56.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.195+0000 7feff184a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:52:56.178 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.195+0000 7feff184a640 10 monclient(hunting): picked mon.noname-a con 0x7fefec1457f0 addr v1:172.21.15.112:6789/0 2024-03-29T10:52:56.178 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.195+0000 7feff184a640 10 monclient(hunting): picked mon.noname-c con 0x7fefec13f650 addr v1:172.21.15.118:6789/0 2024-03-29T10:52:56.178 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.195+0000 7feff184a640 10 monclient(hunting): picked mon.noname-b con 0x7fefec0a9d70 addr v1:172.21.15.112:6790/0 2024-03-29T10:52:56.178 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.195+0000 7feff184a640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefec0aaef0 con 0x7fefec1457f0 2024-03-29T10:52:56.178 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.195+0000 7feff184a640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefec145bd0 con 0x7fefec13f650 2024-03-29T10:52:56.178 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.195+0000 7feff184a640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefec145eb0 con 0x7fefec0a9d70 2024-03-29T10:52:56.178 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.195+0000 7feff184a640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:56.178 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.195+0000 7feff184a640 10 monclient(hunting): authenticate will time out at 931.499584s 2024-03-29T10:52:56.182 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.201+0000 7fefeb7fe640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fefec0a9d70 0x7fefec14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41449/0 (socket says 172.21.15.112:41449) 2024-03-29T10:52:56.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.201+0000 7fefeb7fe640 1 -- 172.21.15.112:0/2388662187 learned_addr learned my addr 172.21.15.112:0/2388662187 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:52:56.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.202+0000 7fefe9ffb640 1 -- 172.21.15.112:0/2388662187 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3793635470 0 0) 0x7fefec0aaef0 con 0x7fefec1457f0 2024-03-29T10:52:56.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.202+0000 7fefe9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:56.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.202+0000 7fefe9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:56.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.202+0000 7fefe9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:56.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.202+0000 7fefe9ffb640 10 monclient(hunting): my global_id is 4281 2024-03-29T10:52:56.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.202+0000 7fefe9ffb640 1 -- 172.21.15.112:0/2388662187 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefd4003520 con 0x7fefec1457f0 2024-03-29T10:52:56.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.202+0000 7fefe9ffb640 1 -- 172.21.15.112:0/2388662187 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3757655771 0 0) 0x7fefd4003520 con 0x7fefec1457f0 2024-03-29T10:52:56.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.202+0000 7fefe9ffb640 1 -- 172.21.15.112:0/2388662187 >> v1:172.21.15.112:6790/0 conn(0x7fefec0a9d70 legacy=0x7fefec14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:56.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.202+0000 7fefe9ffb640 1 -- 172.21.15.112:0/2388662187 >> v1:172.21.15.118:6789/0 conn(0x7fefec13f650 legacy=0x7fefec0b2620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:52:56.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.202+0000 7fefe9ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:52:56.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.202+0000 7fefe9ffb640 1 monclient: found mon.noname-a 2024-03-29T10:52:56.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.202+0000 7fefe9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:56.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.202+0000 7fefe9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T10:52:56.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.202+0000 7fefe9ffb640 1 -- 172.21.15.112:0/2388662187 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefec146340 con 0x7fefec1457f0 2024-03-29T10:52:56.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.202+0000 7fefe9ffb640 10 monclient: _finish_auth 0 2024-03-29T10:52:56.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.202+0000 7fefe9ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:52:56.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.202+0000 7fefe9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:56.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.203+0000 7fefe9ffb640 1 -- 172.21.15.112:0/2388662187 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fefdc001690 con 0x7fefec1457f0 2024-03-29T10:52:56.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.203+0000 7fefe9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:56.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.203+0000 7fefe9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:56.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.203+0000 7fefe9ffb640 10 monclient: dump: 2024-03-29T10:52:56.187 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:56.188 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:56.188 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:56.188 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:56.188 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:56.188 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:56.188 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:56.188 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:56.188 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:56.188 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:56.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.203+0000 7feff184a640 5 monclient: authenticate success, global_id 4281 2024-03-29T10:52:56.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.203+0000 7feff184a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:56.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.203+0000 7fefe9ffb640 1 -- 172.21.15.112:0/2388662187 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefdc001860 con 0x7fefec1457f0 2024-03-29T10:52:56.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.203+0000 7fefe9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:56.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.203+0000 7fefe9ffb640 1 -- 172.21.15.112:0/2388662187 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fefdc001cb0 con 0x7fefec1457f0 2024-03-29T10:52:56.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.203+0000 7fefe9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:56.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.203+0000 7fefe9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:56.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.203+0000 7fefe9ffb640 10 monclient: dump: 2024-03-29T10:52:56.189 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:56.189 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:56.189 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:56.189 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:56.189 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:56.189 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:56.190 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:56.190 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:56.190 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:56.190 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:56.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.203+0000 7feff184a640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:56.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.203+0000 7feff184a640 4 set_mon_vals no callback set 2024-03-29T10:52:56.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.204+0000 7feff184a640 10 monclient: shutdown 2024-03-29T10:52:56.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.204+0000 7feff184a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:56.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.204+0000 7feff184a640 1 -- 172.21.15.112:0/2388662187 >> v1:172.21.15.112:6789/0 conn(0x7fefec1457f0 legacy=0x7fefec13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:56.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.204+0000 7feff184a640 1 -- 172.21.15.112:0/2388662187 shutdown_connections 2024-03-29T10:52:56.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.204+0000 7feff184a640 1 -- 172.21.15.112:0/2388662187 >> 172.21.15.112:0/2388662187 conn(0x7fefec066a00 msgr2=0x7fefec066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:56.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.205+0000 7feff184a640 1 -- 172.21.15.112:0/2388662187 shutdown_connections 2024-03-29T10:52:56.192 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.208+0000 7feff184a640 1 -- 172.21.15.112:0/2388662187 wait complete. 2024-03-29T10:52:56.192 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.208+0000 7feff184a640 10 monclient: build_initial_monmap 2024-03-29T10:52:56.192 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.208+0000 7feff184a640 10 monclient: monmap: 2024-03-29T10:52:56.192 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:52:56.192 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:56.192 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:52:56.192 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:52:56.192 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:56.192 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:56.192 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:52:56.193 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:52:56.193 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:52:56.193 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:56.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.209+0000 7feff184a640 1 librados: starting msgr at 2024-03-29T10:52:56.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.209+0000 7feff184a640 1 librados: starting objecter 2024-03-29T10:52:56.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.209+0000 7feff184a640 1 Processor -- start 2024-03-29T10:52:56.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.209+0000 7feff184a640 1 -- start start 2024-03-29T10:52:56.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.209+0000 7feff184a640 1 librados: setting wanted keys 2024-03-29T10:52:56.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.209+0000 7feff184a640 1 librados: calling monclient init 2024-03-29T10:52:56.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.209+0000 7feff184a640 10 monclient: init 2024-03-29T10:52:56.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.210+0000 7feff184a640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:56.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.210+0000 7feff184a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:52:56.194 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.211+0000 7feff184a640 10 monclient(hunting): picked mon.noname-a con 0x7fefec0a9d70 addr v1:172.21.15.112:6789/0 2024-03-29T10:52:56.194 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.211+0000 7feff184a640 10 monclient(hunting): picked mon.noname-b con 0x7fefec13f650 addr v1:172.21.15.118:6789/0 2024-03-29T10:52:56.195 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.211+0000 7feff184a640 10 monclient(hunting): picked mon.noname-c con 0x7fefec1457f0 addr v1:172.21.15.112:6790/0 2024-03-29T10:52:56.195 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.211+0000 7feff184a640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefec1dfc90 con 0x7fefec0a9d70 2024-03-29T10:52:56.195 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.211+0000 7feff184a640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefec06eda0 con 0x7fefec13f650 2024-03-29T10:52:56.195 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.211+0000 7feff184a640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefec06f080 con 0x7fefec1457f0 2024-03-29T10:52:56.195 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.211+0000 7feff184a640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:56.195 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.211+0000 7feff184a640 10 monclient(hunting): authenticate will time out at 931.515470s 2024-03-29T10:52:56.195 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.213+0000 7fefeaffd640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fefec0a9d70 0x7fefec1dee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:55977/0 (socket says 172.21.15.112:55977) 2024-03-29T10:52:56.195 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.213+0000 7fefeaffd640 1 -- 172.21.15.112:0/1814093685 learned_addr learned my addr 172.21.15.112:0/1814093685 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:52:56.197 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.216+0000 7fefd3fff640 10 client.?.objecter ms_handle_connect 0x7fefec0a9d70 2024-03-29T10:52:56.198 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.216+0000 7fefd3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:56.198 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.216+0000 7fefd3fff640 1 -- 172.21.15.112:0/1814093685 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916288940 0 0) 0x7fefec1dfc90 con 0x7fefec0a9d70 2024-03-29T10:52:56.198 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.216+0000 7fefd3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:56.198 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.216+0000 7fefd3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:56.198 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.216+0000 7fefd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:56.198 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.216+0000 7fefd3fff640 10 monclient(hunting): my global_id is 4284 2024-03-29T10:52:56.198 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.216+0000 7fefd3fff640 1 -- 172.21.15.112:0/1814093685 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefd8004eb0 con 0x7fefec0a9d70 2024-03-29T10:52:56.198 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.217+0000 7fefd3fff640 1 -- 172.21.15.112:0/1814093685 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4001595381 0 0) 0x7fefd8004eb0 con 0x7fefec0a9d70 2024-03-29T10:52:56.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.217+0000 7fefd3fff640 1 -- 172.21.15.112:0/1814093685 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefec1dfc90 con 0x7fefec0a9d70 2024-03-29T10:52:56.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.217+0000 7fefd3fff640 1 -- 172.21.15.112:0/1814093685 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fefdc001c30 con 0x7fefec0a9d70 2024-03-29T10:52:56.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.217+0000 7fefd3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:56.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.217+0000 7fefd3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:56.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.217+0000 7fefd3fff640 10 monclient(hunting): dump: 2024-03-29T10:52:56.199 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:56.199 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:56.199 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:56.199 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:56.199 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:56.199 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:56.199 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:56.199 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:56.200 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:56.200 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:56.200 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.217+0000 7fefd3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:56.201 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.218+0000 7fefd3fff640 1 -- 172.21.15.112:0/1814093685 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 909117196 0 0) 0x7fefec1dfc90 con 0x7fefec0a9d70 2024-03-29T10:52:56.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.218+0000 7fefd3fff640 1 -- 172.21.15.112:0/1814093685 >> v1:172.21.15.112:6790/0 conn(0x7fefec1457f0 legacy=0x7fefec140db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:56.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.218+0000 7fefd3fff640 1 -- 172.21.15.112:0/1814093685 >> v1:172.21.15.118:6789/0 conn(0x7fefec13f650 legacy=0x7fefec1df580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:56.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.218+0000 7fefd3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:52:56.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.218+0000 7fefd3fff640 1 monclient: found mon.a 2024-03-29T10:52:56.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.218+0000 7fefd3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:56.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.218+0000 7fefd3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:52:56.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.218+0000 7fefd3fff640 1 -- 172.21.15.112:0/1814093685 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefec06f360 con 0x7fefec0a9d70 2024-03-29T10:52:56.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.218+0000 7fefd3fff640 10 monclient: _finish_auth 0 2024-03-29T10:52:56.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.218+0000 7fefd3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:52:56.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.218+0000 7fefd3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:56.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.218+0000 7fefd3fff640 1 -- 172.21.15.112:0/1814093685 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefdc0038a0 con 0x7fefec0a9d70 2024-03-29T10:52:56.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.218+0000 7fefd3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:56.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.218+0000 7fefd3fff640 1 -- 172.21.15.112:0/1814093685 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fefdc003cb0 con 0x7fefec0a9d70 2024-03-29T10:52:56.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.218+0000 7fefd3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:56.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.218+0000 7fefd3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:56.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.218+0000 7fefd3fff640 10 monclient: dump: 2024-03-29T10:52:56.203 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:52:56.203 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:52:56.203 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:56.203 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:52:56.203 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:56.203 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:52:56.203 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:52:56.203 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:52:56.203 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:52:56.203 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:52:56.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.218+0000 7fefe8ff9640 4 set_mon_vals no callback set 2024-03-29T10:52:56.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.219+0000 7feff184a640 5 monclient: authenticate success, global_id 4284 2024-03-29T10:52:56.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.219+0000 7feff184a640 10 monclient: _renew_subs 2024-03-29T10:52:56.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.219+0000 7feff184a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:52:56.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.219+0000 7feff184a640 1 -- 172.21.15.112:0/1814093685 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefec06f640 con 0x7fefec0a9d70 2024-03-29T10:52:56.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.219+0000 7feff184a640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:56.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.219+0000 7feff184a640 10 monclient: _renew_subs 2024-03-29T10:52:56.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.219+0000 7feff184a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:52:56.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.219+0000 7feff184a640 1 -- 172.21.15.112:0/1814093685 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fefec06de90 con 0x7fefec0a9d70 2024-03-29T10:52:56.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.219+0000 7feff184a640 1 librados: init done 2024-03-29T10:52:56.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.222+0000 7fefd1ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:56.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.222+0000 7fefd1ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:56.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.222+0000 7fefd1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:52:56.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.222+0000 7fefd1ffb640 1 -- 172.21.15.112:0/1814093685 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefc4005080 con 0x7fefec0a9d70 2024-03-29T10:52:56.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.223+0000 7fefd3fff640 1 -- 172.21.15.112:0/1814093685 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fefdc003e10 con 0x7fefec0a9d70 2024-03-29T10:52:56.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.223+0000 7fefd3fff640 1 -- 172.21.15.112:0/1814093685 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7fefdc084d00 con 0x7fefec0a9d70 2024-03-29T10:52:56.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.223+0000 7fefd3fff640 10 client.4284.objecter ms_dispatch 0x7fefec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:56.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.223+0000 7fefd3fff640 3 client.4284.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:56.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.223+0000 7fefd3fff640 3 client.4284.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:56.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.223+0000 7fefd3fff640 20 client.4284.objecter dump_active .. 0 homeless 2024-03-29T10:52:56.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.225+0000 7fefea7fc640 10 monclient: get_auth_request con 0x7fefd806dc90 auth_method 0 2024-03-29T10:52:56.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.225+0000 7fefd3fff640 10 client.4284.objecter ms_handle_connect 0x7fefd806dc90 2024-03-29T10:52:56.212 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.231+0000 7fefd3fff640 1 -- 172.21.15.112:0/1814093685 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fefdc0505e0 con 0x7fefec0a9d70 2024-03-29T10:52:56.213 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.231+0000 7fefd3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:56.213 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.231+0000 7fefd3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:56.531 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.550+0000 7fefd1ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:56.532 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.550+0000 7fefd1ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:56.532 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.550+0000 7fefd1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:52:56.532 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.550+0000 7fefd1ffb640 1 -- 172.21.15.112:0/1814093685 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fefc4005330 con 0x7fefec0a9d70 2024-03-29T10:52:56.535 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.554+0000 7fefd3fff640 1 -- 172.21.15.112:0/1814093685 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7fefdc052510 con 0x7fefec0a9d70 2024-03-29T10:52:56.535 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.554+0000 7fefd3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:56.535 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.554+0000 7fefd3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:56.537 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:52:56.537 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:56.541 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.559+0000 7fefd1ffb640 10 librados: watch_flush enter 2024-03-29T10:52:56.541 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.559+0000 7fefd1ffb640 10 librados: watch_flush exit 2024-03-29T10:52:56.541 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.559+0000 7fefd1ffb640 20 client.4284.objecter shutdown clearing up homeless session... 2024-03-29T10:52:56.541 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.560+0000 7fefd1ffb640 10 client.4284.objecter successfully canceled tick 2024-03-29T10:52:56.541 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.560+0000 7fefd1ffb640 1 -- 172.21.15.112:0/1814093685 >> v1:172.21.15.118:6816/3181073408 conn(0x7fefd806dc90 legacy=0x7fefd8070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:56.541 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.560+0000 7fefd1ffb640 10 monclient: shutdown 2024-03-29T10:52:56.541 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.560+0000 7fefd1ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:56.541 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.560+0000 7fefd1ffb640 1 -- 172.21.15.112:0/1814093685 >> v1:172.21.15.112:6789/0 conn(0x7fefec0a9d70 legacy=0x7fefec1dee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:56.543 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.561+0000 7fefd1ffb640 1 -- 172.21.15.112:0/1814093685 shutdown_connections 2024-03-29T10:52:56.543 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.561+0000 7fefd1ffb640 1 -- 172.21.15.112:0/1814093685 >> 172.21.15.112:0/1814093685 conn(0x7fefec066a00 msgr2=0x7fefec13aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:56.545 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.564+0000 7fefd1ffb640 1 -- 172.21.15.112:0/1814093685 shutdown_connections 2024-03-29T10:52:56.545 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.564+0000 7fefd1ffb640 1 -- 172.21.15.112:0/1814093685 wait complete. 2024-03-29T10:52:56.545 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:52:56.564+0000 7fefd1ffb640 1 librados: shutdown 2024-03-29T10:52:56.562 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:03.369 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:03.519 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.535+0000 7f64ef40d640 10 monclient: get_monmap_and_config 2024-03-29T10:53:03.519 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.535+0000 7f64ef40d640 10 monclient: build_initial_monmap 2024-03-29T10:53:03.519 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.535+0000 7f64ef40d640 10 monclient: monmap: 2024-03-29T10:53:03.519 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:53:03.519 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:03.519 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:53:03.536191+0000 2024-03-29T10:53:03.519 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:53:03.536191+0000 2024-03-29T10:53:03.519 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:03.519 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:03.519 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:53:03.520 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:53:03.520 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:53:03.520 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:03.520 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.536+0000 7f64ef40d640 1 Processor -- start 2024-03-29T10:53:03.520 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.536+0000 7f64ef40d640 1 -- start start 2024-03-29T10:53:03.520 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.536+0000 7f64ef40d640 10 monclient: init 2024-03-29T10:53:03.520 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.537+0000 7f64ef40d640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:03.520 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.537+0000 7f64ef40d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:03.520 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.537+0000 7f64ef40d640 10 monclient(hunting): picked mon.noname-a con 0x7f64e80ad6c0 addr v1:172.21.15.112:6789/0 2024-03-29T10:53:03.520 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.537+0000 7f64ef40d640 10 monclient(hunting): picked mon.noname-b con 0x7f64e80af6c0 addr v1:172.21.15.112:6790/0 2024-03-29T10:53:03.520 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.537+0000 7f64ef40d640 10 monclient(hunting): picked mon.noname-c con 0x7f64e814fdc0 addr v1:172.21.15.118:6789/0 2024-03-29T10:53:03.520 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.537+0000 7f64ef40d640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e80a94e0 con 0x7f64e80ad6c0 2024-03-29T10:53:03.520 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.537+0000 7f64ef40d640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e80af3a0 con 0x7f64e814fdc0 2024-03-29T10:53:03.520 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.537+0000 7f64ef40d640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e80afb60 con 0x7f64e80af6c0 2024-03-29T10:53:03.521 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.537+0000 7f64ef40d640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:03.521 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.537+0000 7f64ef40d640 10 monclient(hunting): authenticate will time out at 938.841483s 2024-03-29T10:53:03.526 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.545+0000 7f64ed1a9640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f64e80ad6c0 0x7f64e814f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:40567/0 (socket says 172.21.15.112:40567) 2024-03-29T10:53:03.526 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.545+0000 7f64ed1a9640 1 -- 172.21.15.112:0/3958827376 learned_addr learned my addr 172.21.15.112:0/3958827376 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:53:03.529 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.545+0000 7f64dffff640 1 -- 172.21.15.112:0/3958827376 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 505562454 0 0) 0x7f64e80a94e0 con 0x7f64e80ad6c0 2024-03-29T10:53:03.529 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.545+0000 7f64dffff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:03.529 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.545+0000 7f64dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:03.529 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.545+0000 7f64dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:03.529 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.545+0000 7f64dffff640 10 monclient(hunting): my global_id is 4287 2024-03-29T10:53:03.530 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.545+0000 7f64dffff640 1 -- 172.21.15.112:0/3958827376 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64d4003520 con 0x7f64e80ad6c0 2024-03-29T10:53:03.530 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.546+0000 7f64dffff640 1 -- 172.21.15.112:0/3958827376 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280347258 0 0) 0x7f64e80afb60 con 0x7f64e80af6c0 2024-03-29T10:53:03.530 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.546+0000 7f64dffff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:03.530 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.546+0000 7f64dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:03.530 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.546+0000 7f64dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:03.530 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.546+0000 7f64dffff640 10 monclient(hunting): my global_id is 4286 2024-03-29T10:53:03.530 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.546+0000 7f64dffff640 1 -- 172.21.15.112:0/3958827376 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64e80a94e0 con 0x7f64e80af6c0 2024-03-29T10:53:03.530 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.546+0000 7f64dffff640 1 -- 172.21.15.112:0/3958827376 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2977838763 0 0) 0x7f64d4003520 con 0x7f64e80ad6c0 2024-03-29T10:53:03.530 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.546+0000 7f64dffff640 1 -- 172.21.15.112:0/3958827376 >> v1:172.21.15.112:6790/0 conn(0x7f64e80af6c0 legacy=0x7f64e8152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:03.530 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.546+0000 7f64dffff640 1 -- 172.21.15.112:0/3958827376 >> v1:172.21.15.118:6789/0 conn(0x7f64e814fdc0 legacy=0x7f64e815a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:53:03.530 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.546+0000 7f64dffff640 10 monclient: _finish_hunting 0 2024-03-29T10:53:03.530 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.546+0000 7f64dffff640 1 monclient: found mon.noname-a 2024-03-29T10:53:03.530 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.546+0000 7f64dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:03.530 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.547+0000 7f64dffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T10:53:03.531 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.547+0000 7f64dffff640 1 -- 172.21.15.112:0/3958827376 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e80aab80 con 0x7f64e80ad6c0 2024-03-29T10:53:03.531 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.547+0000 7f64dffff640 10 monclient: _finish_auth 0 2024-03-29T10:53:03.531 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.547+0000 7f64dffff640 10 monclient: _check_auth_tickets 2024-03-29T10:53:03.531 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.547+0000 7f64dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:03.531 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.547+0000 7f64dffff640 1 -- 172.21.15.112:0/3958827376 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f64e4001690 con 0x7f64e80ad6c0 2024-03-29T10:53:03.531 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.547+0000 7f64dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:03.531 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.547+0000 7f64dffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:03.531 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.547+0000 7f64dffff640 10 monclient: dump: 2024-03-29T10:53:03.531 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:03.532 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:03.532 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:03.532 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:03.532 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:03.532 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:03.532 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:03.532 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:03.532 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:03.532 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:03.532 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.547+0000 7f64ef40d640 5 monclient: authenticate success, global_id 4287 2024-03-29T10:53:03.532 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.547+0000 7f64ef40d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:03.532 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.548+0000 7f64dffff640 1 -- 172.21.15.112:0/3958827376 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64e40017f0 con 0x7f64e80ad6c0 2024-03-29T10:53:03.532 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.548+0000 7f64dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:03.533 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.548+0000 7f64dffff640 1 -- 172.21.15.112:0/3958827376 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f64e4001c00 con 0x7f64e80ad6c0 2024-03-29T10:53:03.533 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.548+0000 7f64dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:03.534 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.548+0000 7f64dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:03.534 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.548+0000 7f64dffff640 10 monclient: dump: 2024-03-29T10:53:03.534 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:03.534 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:03.534 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:03.534 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:03.534 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:03.534 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:03.534 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:03.534 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:03.534 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:03.534 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:03.534 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.548+0000 7f64ef40d640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:03.534 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.548+0000 7f64ef40d640 4 set_mon_vals no callback set 2024-03-29T10:53:03.534 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.549+0000 7f64ef40d640 10 monclient: shutdown 2024-03-29T10:53:03.535 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.549+0000 7f64ef40d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:03.535 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.549+0000 7f64ef40d640 1 -- 172.21.15.112:0/3958827376 >> v1:172.21.15.112:6789/0 conn(0x7f64e80ad6c0 legacy=0x7f64e814f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:03.535 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.552+0000 7f64ef40d640 1 -- 172.21.15.112:0/3958827376 shutdown_connections 2024-03-29T10:53:03.535 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.552+0000 7f64ef40d640 1 -- 172.21.15.112:0/3958827376 >> 172.21.15.112:0/3958827376 conn(0x7f64e80b35e0 msgr2=0x7f64e80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:03.536 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.555+0000 7f64ef40d640 1 -- 172.21.15.112:0/3958827376 shutdown_connections 2024-03-29T10:53:03.537 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.555+0000 7f64ef40d640 1 -- 172.21.15.112:0/3958827376 wait complete. 2024-03-29T10:53:03.538 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.556+0000 7f64ef40d640 10 monclient: build_initial_monmap 2024-03-29T10:53:03.538 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.556+0000 7f64ef40d640 10 monclient: monmap: 2024-03-29T10:53:03.538 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:53:03.538 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:03.538 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:53:03.538 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:53:03.538 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:03.538 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:03.539 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:53:03.539 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:53:03.539 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:53:03.539 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:03.539 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.556+0000 7f64ef40d640 1 librados: starting msgr at 2024-03-29T10:53:03.539 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.556+0000 7f64ef40d640 1 librados: starting objecter 2024-03-29T10:53:03.539 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.556+0000 7f64ef40d640 1 Processor -- start 2024-03-29T10:53:03.539 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.557+0000 7f64ef40d640 1 -- start start 2024-03-29T10:53:03.539 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.557+0000 7f64ef40d640 1 librados: setting wanted keys 2024-03-29T10:53:03.539 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.557+0000 7f64ef40d640 1 librados: calling monclient init 2024-03-29T10:53:03.539 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.557+0000 7f64ef40d640 10 monclient: init 2024-03-29T10:53:03.540 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.557+0000 7f64ef40d640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:03.540 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.557+0000 7f64ef40d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:53:03.541 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.557+0000 7f64ef40d640 10 monclient(hunting): picked mon.noname-c con 0x7f64e80ad6c0 addr v1:172.21.15.112:6790/0 2024-03-29T10:53:03.541 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.557+0000 7f64ef40d640 10 monclient(hunting): picked mon.noname-a con 0x7f64e80af6c0 addr v1:172.21.15.112:6789/0 2024-03-29T10:53:03.541 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.557+0000 7f64ef40d640 10 monclient(hunting): picked mon.noname-b con 0x7f64e814fdc0 addr v1:172.21.15.118:6789/0 2024-03-29T10:53:03.541 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.557+0000 7f64ef40d640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e8153690 con 0x7f64e80af6c0 2024-03-29T10:53:03.541 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.557+0000 7f64ef40d640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e8153970 con 0x7f64e814fdc0 2024-03-29T10:53:03.541 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.557+0000 7f64ef40d640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e8071bd0 con 0x7f64e80ad6c0 2024-03-29T10:53:03.541 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.557+0000 7f64ef40d640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:03.541 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.557+0000 7f64ef40d640 10 monclient(hunting): authenticate will time out at 938.862152s 2024-03-29T10:53:03.542 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.561+0000 7f64ed9aa640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f64e814fdc0 0x7f64e81f8db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:58815/0 (socket says 172.21.15.112:58815) 2024-03-29T10:53:03.542 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.561+0000 7f64ed9aa640 1 -- 172.21.15.112:0/4058173060 learned_addr learned my addr 172.21.15.112:0/4058173060 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:53:03.545 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.563+0000 7f64ddffb640 10 client.?.objecter ms_handle_connect 0x7f64e814fdc0 2024-03-29T10:53:03.545 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.563+0000 7f64ddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:03.546 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.565+0000 7f64ddffb640 1 -- 172.21.15.112:0/4058173060 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3895829364 0 0) 0x7f64e8153970 con 0x7f64e814fdc0 2024-03-29T10:53:03.547 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.565+0000 7f64ddffb640 10 monclient(hunting): _negotiate 2024-03-29T10:53:03.547 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.565+0000 7f64ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:03.547 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.565+0000 7f64ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:03.547 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.565+0000 7f64ddffb640 10 monclient(hunting): my global_id is 4294 2024-03-29T10:53:03.547 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.565+0000 7f64ddffb640 1 -- 172.21.15.112:0/4058173060 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64cc004eb0 con 0x7f64e814fdc0 2024-03-29T10:53:03.547 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.565+0000 7f64ddffb640 10 client.?.objecter ms_handle_connect 0x7f64e80ad6c0 2024-03-29T10:53:03.547 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.565+0000 7f64ddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:03.550 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.566+0000 7f64ddffb640 1 -- 172.21.15.112:0/4058173060 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3520326033 0 0) 0x7f64e8071bd0 con 0x7f64e80ad6c0 2024-03-29T10:53:03.550 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.566+0000 7f64ddffb640 10 monclient(hunting): _negotiate 2024-03-29T10:53:03.550 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.566+0000 7f64ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:03.550 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.566+0000 7f64ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:03.550 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.566+0000 7f64ddffb640 10 monclient(hunting): my global_id is 4289 2024-03-29T10:53:03.550 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.566+0000 7f64ddffb640 1 -- 172.21.15.112:0/4058173060 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64e8153970 con 0x7f64e80ad6c0 2024-03-29T10:53:03.550 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.566+0000 7f64ddffb640 1 -- 172.21.15.112:0/4058173060 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 34245649 0 0) 0x7f64cc004eb0 con 0x7f64e814fdc0 2024-03-29T10:53:03.550 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.566+0000 7f64ddffb640 1 -- 172.21.15.112:0/4058173060 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64e8071bd0 con 0x7f64e814fdc0 2024-03-29T10:53:03.550 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.566+0000 7f64ddffb640 1 -- 172.21.15.112:0/4058173060 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f64e00042e0 con 0x7f64e814fdc0 2024-03-29T10:53:03.551 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.566+0000 7f64ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:03.551 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.566+0000 7f64ddffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:03.551 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.566+0000 7f64ddffb640 10 monclient(hunting): dump: 2024-03-29T10:53:03.551 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:03.551 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:03.551 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:03.551 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:03.551 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:03.551 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:03.551 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:03.551 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:03.551 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:03.551 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:03.551 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.566+0000 7f64ddffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:03.552 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.567+0000 7f64ddffb640 10 client.?.objecter ms_handle_connect 0x7f64e80af6c0 2024-03-29T10:53:03.552 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.567+0000 7f64ddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:03.555 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.569+0000 7f64ddffb640 1 -- 172.21.15.112:0/4058173060 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4251547719 0 0) 0x7f64e8153970 con 0x7f64e80ad6c0 2024-03-29T10:53:03.555 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.569+0000 7f64ddffb640 1 -- 172.21.15.112:0/4058173060 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64cc004eb0 con 0x7f64e80ad6c0 2024-03-29T10:53:03.555 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.569+0000 7f64ddffb640 1 -- 172.21.15.112:0/4058173060 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f64e4004960 con 0x7f64e80ad6c0 2024-03-29T10:53:03.555 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.569+0000 7f64ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:03.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.569+0000 7f64ddffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:03.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.569+0000 7f64ddffb640 10 monclient(hunting): dump: 2024-03-29T10:53:03.556 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:03.556 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:03.556 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:03.556 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:03.556 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:03.556 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:03.556 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:03.556 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:03.556 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:03.556 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:03.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.569+0000 7f64ddffb640 1 -- 172.21.15.112:0/4058173060 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 88865518 0 0) 0x7f64e8071bd0 con 0x7f64e814fdc0 2024-03-29T10:53:03.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.569+0000 7f64ddffb640 1 -- 172.21.15.112:0/4058173060 >> v1:172.21.15.112:6790/0 conn(0x7f64e80ad6c0 legacy=0x7f64e81f22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:03.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.569+0000 7f64ddffb640 1 -- 172.21.15.112:0/4058173060 >> v1:172.21.15.112:6789/0 conn(0x7f64e80af6c0 legacy=0x7f64e81f86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:03.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.569+0000 7f64ddffb640 10 monclient: _finish_hunting 0 2024-03-29T10:53:03.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.569+0000 7f64ddffb640 1 monclient: found mon.b 2024-03-29T10:53:03.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.569+0000 7f64ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:03.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.569+0000 7f64ddffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:53:03.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.569+0000 7f64ddffb640 1 -- 172.21.15.112:0/4058173060 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e8153d00 con 0x7f64e814fdc0 2024-03-29T10:53:03.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.569+0000 7f64ddffb640 10 monclient: _finish_auth 0 2024-03-29T10:53:03.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.569+0000 7f64ddffb640 10 monclient: _check_auth_tickets 2024-03-29T10:53:03.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.569+0000 7f64ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:03.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.570+0000 7f64ef40d640 5 monclient: authenticate success, global_id 4294 2024-03-29T10:53:03.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.570+0000 7f64ef40d640 10 monclient: _renew_subs 2024-03-29T10:53:03.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.570+0000 7f64ef40d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:53:03.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.570+0000 7f64ef40d640 1 -- 172.21.15.112:0/4058173060 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e80723b0 con 0x7f64e814fdc0 2024-03-29T10:53:03.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.570+0000 7f64ef40d640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:03.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.570+0000 7f64ef40d640 10 monclient: _renew_subs 2024-03-29T10:53:03.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.570+0000 7f64ef40d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:53:03.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.570+0000 7f64ef40d640 1 -- 172.21.15.112:0/4058173060 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e81f2bd0 con 0x7f64e814fdc0 2024-03-29T10:53:03.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.570+0000 7f64ef40d640 1 librados: init done 2024-03-29T10:53:03.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.571+0000 7f64ef40d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:03.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.571+0000 7f64ef40d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:03.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.571+0000 7f64ef40d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:53:03.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.571+0000 7f64ef40d640 1 -- 172.21.15.112:0/4058173060 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e8153e90 con 0x7f64e814fdc0 2024-03-29T10:53:03.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.572+0000 7f64ddffb640 1 -- 172.21.15.112:0/4058173060 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64e00047f0 con 0x7f64e814fdc0 2024-03-29T10:53:03.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.572+0000 7f64ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:03.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.572+0000 7f64df7fe640 4 set_mon_vals no callback set 2024-03-29T10:53:03.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.573+0000 7f64ddffb640 1 -- 172.21.15.112:0/4058173060 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f64e00013f0 con 0x7f64e814fdc0 2024-03-29T10:53:03.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.573+0000 7f64ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:03.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.573+0000 7f64ddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:03.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.573+0000 7f64ddffb640 10 monclient: dump: 2024-03-29T10:53:03.558 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:03.559 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:03.559 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:03.559 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:03.559 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:03.559 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:03.559 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:03.559 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:03.559 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:03.559 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:03.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.574+0000 7f64ddffb640 1 -- 172.21.15.112:0/4058173060 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f64e0001610 con 0x7f64e814fdc0 2024-03-29T10:53:03.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.575+0000 7f64ddffb640 1 -- 172.21.15.112:0/4058173060 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f64e0084c40 con 0x7f64e814fdc0 2024-03-29T10:53:03.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.575+0000 7f64ddffb640 10 client.4294.objecter ms_dispatch 0x7f64e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:03.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.575+0000 7f64ddffb640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:03.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.575+0000 7f64ddffb640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:03.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.575+0000 7f64ddffb640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-29T10:53:03.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.576+0000 7f64ed1a9640 10 monclient: get_auth_request con 0x7f64cc06de90 auth_method 0 2024-03-29T10:53:03.561 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.576+0000 7f64ddffb640 10 client.4294.objecter ms_handle_connect 0x7f64cc06de90 2024-03-29T10:53:03.561 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.580+0000 7f64ddffb640 1 -- 172.21.15.112:0/4058173060 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f64e0050520 con 0x7f64e814fdc0 2024-03-29T10:53:03.562 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.580+0000 7f64ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:03.562 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.580+0000 7f64ddffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:03.885 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.904+0000 7f64ef40d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:03.886 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.904+0000 7f64ef40d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:03.886 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.904+0000 7f64ef40d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:53:03.886 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.904+0000 7f64ef40d640 1 -- 172.21.15.112:0/4058173060 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64e8153b20 con 0x7f64e814fdc0 2024-03-29T10:53:03.889 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.908+0000 7f64ddffb640 1 -- 172.21.15.112:0/4058173060 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f64e8153b20 con 0x7f64e814fdc0 2024-03-29T10:53:03.889 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.908+0000 7f64ddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:03.889 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.908+0000 7f64ddffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:03.891 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:53:03.892 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:03.895 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.911+0000 7f64ef40d640 10 librados: watch_flush enter 2024-03-29T10:53:03.895 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.911+0000 7f64ef40d640 10 librados: watch_flush exit 2024-03-29T10:53:03.895 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.911+0000 7f64ef40d640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-29T10:53:03.895 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.911+0000 7f64ef40d640 10 client.4294.objecter successfully canceled tick 2024-03-29T10:53:03.895 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.913+0000 7f64ef40d640 1 -- 172.21.15.112:0/4058173060 >> v1:172.21.15.118:6816/3181073408 conn(0x7f64cc06de90 legacy=0x7f64cc070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:03.895 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.913+0000 7f64ef40d640 10 monclient: shutdown 2024-03-29T10:53:03.895 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.913+0000 7f64ef40d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:03.895 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.913+0000 7f64ef40d640 1 -- 172.21.15.112:0/4058173060 >> v1:172.21.15.118:6789/0 conn(0x7f64e814fdc0 legacy=0x7f64e81f8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:03.895 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.913+0000 7f64ef40d640 1 -- 172.21.15.112:0/4058173060 shutdown_connections 2024-03-29T10:53:03.896 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.913+0000 7f64ef40d640 1 -- 172.21.15.112:0/4058173060 >> 172.21.15.112:0/4058173060 conn(0x7f64e80b35e0 msgr2=0x7f64e80b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:03.896 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.913+0000 7f64ef40d640 1 -- 172.21.15.112:0/4058173060 shutdown_connections 2024-03-29T10:53:03.896 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.914+0000 7f64ef40d640 1 -- 172.21.15.112:0/4058173060 wait complete. 2024-03-29T10:53:03.898 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:03.917+0000 7f64ef40d640 1 librados: shutdown 2024-03-29T10:53:03.915 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:10.722 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:10.872 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.891+0000 7f0d7e56b640 10 monclient: get_monmap_and_config 2024-03-29T10:53:10.872 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.891+0000 7f0d7e56b640 10 monclient: build_initial_monmap 2024-03-29T10:53:10.872 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.891+0000 7f0d7e56b640 10 monclient: monmap: 2024-03-29T10:53:10.872 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:53:10.872 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:10.872 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:53:10.892030+0000 2024-03-29T10:53:10.873 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:53:10.892030+0000 2024-03-29T10:53:10.873 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:10.873 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:10.873 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:53:10.873 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:53:10.873 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:53:10.873 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:10.875 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.893+0000 7f0d7e56b640 1 Processor -- start 2024-03-29T10:53:10.875 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.893+0000 7f0d7e56b640 1 -- start start 2024-03-29T10:53:10.875 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.893+0000 7f0d7e56b640 10 monclient: init 2024-03-29T10:53:10.876 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.894+0000 7f0d7e56b640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:10.876 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.894+0000 7f0d7e56b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:10.876 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.894+0000 7f0d7e56b640 10 monclient(hunting): picked mon.noname-a con 0x7f0d780af2e0 addr v1:172.21.15.112:6789/0 2024-03-29T10:53:10.876 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.894+0000 7f0d7e56b640 10 monclient(hunting): picked mon.noname-b con 0x7f0d7806d8c0 addr v1:172.21.15.112:6790/0 2024-03-29T10:53:10.876 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.894+0000 7f0d7e56b640 10 monclient(hunting): picked mon.noname-c con 0x7f0d780a96c0 addr v1:172.21.15.118:6789/0 2024-03-29T10:53:10.876 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.894+0000 7f0d7e56b640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d780a9b50 con 0x7f0d780af2e0 2024-03-29T10:53:10.876 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.894+0000 7f0d7e56b640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d780ac4f0 con 0x7f0d780a96c0 2024-03-29T10:53:10.877 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.894+0000 7f0d7e56b640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d780af770 con 0x7f0d7806d8c0 2024-03-29T10:53:10.877 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.894+0000 7f0d7e56b640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:10.877 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.894+0000 7f0d7e56b640 10 monclient(hunting): authenticate will time out at 946.199052s 2024-03-29T10:53:10.884 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.902+0000 7f0d77fff640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f0d780af2e0 0x7f0d7813b4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59337/0 (socket says 172.21.15.112:59337) 2024-03-29T10:53:10.884 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.903+0000 7f0d77fff640 1 -- 172.21.15.112:0/3363864372 learned_addr learned my addr 172.21.15.112:0/3363864372 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:53:10.885 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.903+0000 7f0d76ffd640 1 -- 172.21.15.112:0/3363864372 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052188157 0 0) 0x7f0d780a9b50 con 0x7f0d780af2e0 2024-03-29T10:53:10.885 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.903+0000 7f0d76ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:10.885 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.903+0000 7f0d76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:10.885 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.903+0000 7f0d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:10.885 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.903+0000 7f0d76ffd640 10 monclient(hunting): my global_id is 4290 2024-03-29T10:53:10.885 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.903+0000 7f0d76ffd640 1 -- 172.21.15.112:0/3363864372 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d60003520 con 0x7f0d780af2e0 2024-03-29T10:53:10.886 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.903+0000 7f0d76ffd640 1 -- 172.21.15.112:0/3363864372 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1168904439 0 0) 0x7f0d60003520 con 0x7f0d780af2e0 2024-03-29T10:53:10.886 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.903+0000 7f0d76ffd640 1 -- 172.21.15.112:0/3363864372 >> v1:172.21.15.112:6790/0 conn(0x7f0d7806d8c0 legacy=0x7f0d781417e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:53:10.886 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.903+0000 7f0d76ffd640 1 -- 172.21.15.112:0/3363864372 >> v1:172.21.15.118:6789/0 conn(0x7f0d780a96c0 legacy=0x7f0d78141ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:53:10.886 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.903+0000 7f0d76ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:53:10.886 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.904+0000 7f0d76ffd640 1 monclient: found mon.noname-a 2024-03-29T10:53:10.887 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.904+0000 7f0d76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:10.887 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.904+0000 7f0d76ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T10:53:10.887 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.904+0000 7f0d76ffd640 1 -- 172.21.15.112:0/3363864372 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d780b00c0 con 0x7f0d780af2e0 2024-03-29T10:53:10.887 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.904+0000 7f0d76ffd640 10 monclient: _finish_auth 0 2024-03-29T10:53:10.887 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.904+0000 7f0d76ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:53:10.887 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.904+0000 7f0d76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:10.887 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.904+0000 7f0d76ffd640 1 -- 172.21.15.112:0/3363864372 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f0d68001690 con 0x7f0d780af2e0 2024-03-29T10:53:10.887 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.904+0000 7f0d76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:10.887 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.904+0000 7f0d76ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:10.887 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.904+0000 7f0d76ffd640 10 monclient: dump: 2024-03-29T10:53:10.888 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:10.888 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:10.888 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:10.888 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:10.888 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:10.888 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:10.888 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:10.888 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:10.888 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:10.888 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:10.888 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.904+0000 7f0d76ffd640 1 -- 172.21.15.112:0/3363864372 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d680018b0 con 0x7f0d780af2e0 2024-03-29T10:53:10.888 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.904+0000 7f0d76ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:10.889 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.904+0000 7f0d76ffd640 1 -- 172.21.15.112:0/3363864372 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f0d68001c40 con 0x7f0d780af2e0 2024-03-29T10:53:10.889 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.904+0000 7f0d76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:10.889 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.904+0000 7f0d76ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:10.889 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.904+0000 7f0d76ffd640 10 monclient: dump: 2024-03-29T10:53:10.889 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:10.889 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:10.889 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:10.890 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:10.890 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:10.890 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:10.890 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:10.890 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:10.890 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:10.890 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:10.890 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.904+0000 7f0d7e56b640 5 monclient: authenticate success, global_id 4290 2024-03-29T10:53:10.890 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.905+0000 7f0d7e56b640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:10.890 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.905+0000 7f0d7e56b640 4 set_mon_vals no callback set 2024-03-29T10:53:10.894 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.905+0000 7f0d7e56b640 10 monclient: shutdown 2024-03-29T10:53:10.894 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.906+0000 7f0d7e56b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:10.894 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.906+0000 7f0d7e56b640 1 -- 172.21.15.112:0/3363864372 >> v1:172.21.15.112:6789/0 conn(0x7f0d780af2e0 legacy=0x7f0d7813b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:10.894 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.906+0000 7f0d7e56b640 1 -- 172.21.15.112:0/3363864372 shutdown_connections 2024-03-29T10:53:10.894 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.906+0000 7f0d7e56b640 1 -- 172.21.15.112:0/3363864372 >> 172.21.15.112:0/3363864372 conn(0x7f0d780b3a00 msgr2=0x7f0d780b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:10.894 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.907+0000 7f0d7e56b640 1 -- 172.21.15.112:0/3363864372 shutdown_connections 2024-03-29T10:53:10.894 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.907+0000 7f0d7e56b640 1 -- 172.21.15.112:0/3363864372 wait complete. 2024-03-29T10:53:10.895 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.908+0000 7f0d7e56b640 10 monclient: build_initial_monmap 2024-03-29T10:53:10.895 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.908+0000 7f0d7e56b640 10 monclient: monmap: 2024-03-29T10:53:10.895 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:53:10.895 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:10.895 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:53:10.895 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:53:10.895 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:10.895 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:10.895 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:53:10.895 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:53:10.896 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:53:10.896 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:10.896 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.909+0000 7f0d7e56b640 1 librados: starting msgr at 2024-03-29T10:53:10.896 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.909+0000 7f0d7e56b640 1 librados: starting objecter 2024-03-29T10:53:10.896 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.909+0000 7f0d7e56b640 1 Processor -- start 2024-03-29T10:53:10.896 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.909+0000 7f0d7e56b640 1 -- start start 2024-03-29T10:53:10.896 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.909+0000 7f0d7e56b640 1 librados: setting wanted keys 2024-03-29T10:53:10.896 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.909+0000 7f0d7e56b640 1 librados: calling monclient init 2024-03-29T10:53:10.896 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.909+0000 7f0d7e56b640 10 monclient: init 2024-03-29T10:53:10.896 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.910+0000 7f0d7e56b640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:10.897 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.910+0000 7f0d7e56b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:10.897 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.910+0000 7f0d7e56b640 10 monclient(hunting): picked mon.noname-a con 0x7f0d7806d8c0 addr v1:172.21.15.112:6789/0 2024-03-29T10:53:10.897 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.910+0000 7f0d7e56b640 10 monclient(hunting): picked mon.noname-c con 0x7f0d780a96c0 addr v1:172.21.15.112:6790/0 2024-03-29T10:53:10.897 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.910+0000 7f0d7e56b640 10 monclient(hunting): picked mon.noname-b con 0x7f0d7814c3e0 addr v1:172.21.15.118:6789/0 2024-03-29T10:53:10.897 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.910+0000 7f0d7e56b640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d78135d90 con 0x7f0d7806d8c0 2024-03-29T10:53:10.897 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.911+0000 7f0d7e56b640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d7813ed40 con 0x7f0d7814c3e0 2024-03-29T10:53:10.897 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.911+0000 7f0d7e56b640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d7813bea0 con 0x7f0d780a96c0 2024-03-29T10:53:10.897 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.911+0000 7f0d7e56b640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:10.897 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.911+0000 7f0d7e56b640 10 monclient(hunting): authenticate will time out at 946.215363s 2024-03-29T10:53:10.900 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.916+0000 7f0d77fff640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f0d7806d8c0 0x7f0d7813c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59537/0 (socket says 172.21.15.112:59537) 2024-03-29T10:53:10.900 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.916+0000 7f0d77fff640 1 -- 172.21.15.112:0/2821352353 learned_addr learned my addr 172.21.15.112:0/2821352353 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:53:10.900 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.916+0000 7f0d74ff9640 10 client.?.objecter ms_handle_connect 0x7f0d7806d8c0 2024-03-29T10:53:10.900 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.916+0000 7f0d74ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:10.900 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.916+0000 7f0d74ff9640 1 -- 172.21.15.112:0/2821352353 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813959938 0 0) 0x7f0d78135d90 con 0x7f0d7806d8c0 2024-03-29T10:53:10.900 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.916+0000 7f0d74ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:53:10.900 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.916+0000 7f0d74ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:10.901 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.916+0000 7f0d74ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:10.901 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.916+0000 7f0d74ff9640 10 monclient(hunting): my global_id is 4293 2024-03-29T10:53:10.901 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.916+0000 7f0d74ff9640 1 -- 172.21.15.112:0/2821352353 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d64004eb0 con 0x7f0d7806d8c0 2024-03-29T10:53:10.901 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.916+0000 7f0d74ff9640 1 -- 172.21.15.112:0/2821352353 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1318232082 0 0) 0x7f0d64004eb0 con 0x7f0d7806d8c0 2024-03-29T10:53:10.901 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.916+0000 7f0d74ff9640 1 -- 172.21.15.112:0/2821352353 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d64005300 con 0x7f0d7806d8c0 2024-03-29T10:53:10.901 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.916+0000 7f0d74ff9640 1 -- 172.21.15.112:0/2821352353 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f0d68001550 con 0x7f0d7806d8c0 2024-03-29T10:53:10.901 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.916+0000 7f0d74ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:10.901 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.916+0000 7f0d74ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:10.901 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.916+0000 7f0d74ff9640 10 monclient(hunting): dump: 2024-03-29T10:53:10.901 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:10.902 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:10.902 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:10.902 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:10.902 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:10.902 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:10.902 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:10.902 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:10.902 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:10.902 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:10.902 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.916+0000 7f0d74ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:10.902 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.917+0000 7f0d74ff9640 1 -- 172.21.15.112:0/2821352353 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 535385436 0 0) 0x7f0d64005300 con 0x7f0d7806d8c0 2024-03-29T10:53:10.902 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.917+0000 7f0d74ff9640 1 -- 172.21.15.112:0/2821352353 >> v1:172.21.15.112:6790/0 conn(0x7f0d780a96c0 legacy=0x7f0d7814bcd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:53:10.904 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.921+0000 7f0d74ff9640 1 -- 172.21.15.112:0/2821352353 >> v1:172.21.15.118:6789/0 conn(0x7f0d7814c3e0 legacy=0x7f0d781db390 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:53:10.905 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.921+0000 7f0d74ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:53:10.905 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.921+0000 7f0d74ff9640 1 monclient: found mon.a 2024-03-29T10:53:10.905 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.921+0000 7f0d74ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:10.905 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.921+0000 7f0d74ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:53:10.905 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.921+0000 7f0d74ff9640 1 -- 172.21.15.112:0/2821352353 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7813d4d0 con 0x7f0d7806d8c0 2024-03-29T10:53:10.905 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.921+0000 7f0d74ff9640 10 monclient: _finish_auth 0 2024-03-29T10:53:10.905 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.921+0000 7f0d74ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:53:10.905 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.921+0000 7f0d74ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:10.905 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.922+0000 7f0d7e56b640 5 monclient: authenticate success, global_id 4293 2024-03-29T10:53:10.905 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.922+0000 7f0d7e56b640 10 monclient: _renew_subs 2024-03-29T10:53:10.905 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.922+0000 7f0d7e56b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:53:10.905 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.922+0000 7f0d7e56b640 1 -- 172.21.15.112:0/2821352353 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d7813d6c0 con 0x7f0d7806d8c0 2024-03-29T10:53:10.905 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.922+0000 7f0d74ff9640 1 -- 172.21.15.112:0/2821352353 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d68004bf0 con 0x7f0d7806d8c0 2024-03-29T10:53:10.906 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.922+0000 7f0d74ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:10.906 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.922+0000 7f0d74ff9640 1 -- 172.21.15.112:0/2821352353 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f0d68004f00 con 0x7f0d7806d8c0 2024-03-29T10:53:10.906 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.922+0000 7f0d74ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:10.908 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.922+0000 7f0d74ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:10.908 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.922+0000 7f0d74ff9640 10 monclient: dump: 2024-03-29T10:53:10.908 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:10.908 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:10.908 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:10.908 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:10.908 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:10.908 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:10.909 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:10.909 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:10.909 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:10.909 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:10.909 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.923+0000 7f0d7e56b640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:10.909 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.923+0000 7f0d7e56b640 10 monclient: _renew_subs 2024-03-29T10:53:10.909 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.923+0000 7f0d7e56b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:53:10.909 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.923+0000 7f0d7e56b640 1 -- 172.21.15.112:0/2821352353 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d781dbc80 con 0x7f0d7806d8c0 2024-03-29T10:53:10.909 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.923+0000 7f0d7e56b640 1 librados: init done 2024-03-29T10:53:10.909 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.924+0000 7f0d7e56b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:10.909 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.924+0000 7f0d7e56b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:10.910 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.924+0000 7f0d7e56b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:53:10.910 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.924+0000 7f0d7e56b640 1 -- 172.21.15.112:0/2821352353 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d780ac210 con 0x7f0d7806d8c0 2024-03-29T10:53:10.910 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.926+0000 7f0d75ffb640 4 set_mon_vals no callback set 2024-03-29T10:53:10.911 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.927+0000 7f0d74ff9640 1 -- 172.21.15.112:0/2821352353 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f0d6801aed0 con 0x7f0d7806d8c0 2024-03-29T10:53:10.911 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.928+0000 7f0d74ff9640 1 -- 172.21.15.112:0/2821352353 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f0d68084ba0 con 0x7f0d7806d8c0 2024-03-29T10:53:10.911 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.928+0000 7f0d74ff9640 10 client.4293.objecter ms_dispatch 0x7f0d78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:10.911 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.928+0000 7f0d74ff9640 3 client.4293.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:10.911 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.928+0000 7f0d74ff9640 3 client.4293.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:10.911 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.928+0000 7f0d74ff9640 20 client.4293.objecter dump_active .. 0 homeless 2024-03-29T10:53:10.911 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.930+0000 7f0d777fe640 10 monclient: get_auth_request con 0x7f0d64076680 auth_method 0 2024-03-29T10:53:10.912 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.930+0000 7f0d74ff9640 10 client.4293.objecter ms_handle_connect 0x7f0d64076680 2024-03-29T10:53:10.916 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.935+0000 7f0d74ff9640 1 -- 172.21.15.112:0/2821352353 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0d68050480 con 0x7f0d7806d8c0 2024-03-29T10:53:10.916 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.935+0000 7f0d74ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:10.916 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:10.935+0000 7f0d74ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:11.241 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:11.259+0000 7f0d7e56b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:11.241 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:11.259+0000 7f0d7e56b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:11.241 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:11.259+0000 7f0d7e56b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:53:11.241 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:11.259+0000 7f0d7e56b640 1 -- 172.21.15.112:0/2821352353 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d7804f730 con 0x7f0d7806d8c0 2024-03-29T10:53:11.242 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:11.260+0000 7f0d74ff9640 1 -- 172.21.15.112:0/2821352353 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f0d68050480 con 0x7f0d7806d8c0 2024-03-29T10:53:11.242 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:11.260+0000 7f0d74ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:11.242 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:11.260+0000 7f0d74ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:11.242 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:53:11.242 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:11.248 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:11.264+0000 7f0d7e56b640 10 librados: watch_flush enter 2024-03-29T10:53:11.248 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:11.264+0000 7f0d7e56b640 10 librados: watch_flush exit 2024-03-29T10:53:11.248 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:11.264+0000 7f0d7e56b640 20 client.4293.objecter shutdown clearing up homeless session... 2024-03-29T10:53:11.248 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:11.264+0000 7f0d7e56b640 10 client.4293.objecter successfully canceled tick 2024-03-29T10:53:11.248 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:11.265+0000 7f0d7e56b640 1 -- 172.21.15.112:0/2821352353 >> v1:172.21.15.118:6816/3181073408 conn(0x7f0d64076680 legacy=0x7f0d64078b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:11.248 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:11.265+0000 7f0d7e56b640 10 monclient: shutdown 2024-03-29T10:53:11.248 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:11.265+0000 7f0d7e56b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:11.249 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:11.265+0000 7f0d7e56b640 1 -- 172.21.15.112:0/2821352353 >> v1:172.21.15.112:6789/0 conn(0x7f0d7806d8c0 legacy=0x7f0d7813c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:11.249 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:11.266+0000 7f0d7e56b640 1 -- 172.21.15.112:0/2821352353 shutdown_connections 2024-03-29T10:53:11.249 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:11.266+0000 7f0d7e56b640 1 -- 172.21.15.112:0/2821352353 >> 172.21.15.112:0/2821352353 conn(0x7f0d780b3a00 msgr2=0x7f0d780b02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:11.249 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:11.267+0000 7f0d7e56b640 1 -- 172.21.15.112:0/2821352353 shutdown_connections 2024-03-29T10:53:11.249 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:11.267+0000 7f0d7e56b640 1 -- 172.21.15.112:0/2821352353 wait complete. 2024-03-29T10:53:11.249 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:11.267+0000 7f0d7e56b640 1 librados: shutdown 2024-03-29T10:53:11.269 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:18.077 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:18.230 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.247+0000 7fc19266b640 10 monclient: get_monmap_and_config 2024-03-29T10:53:18.230 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.248+0000 7fc19266b640 10 monclient: build_initial_monmap 2024-03-29T10:53:18.230 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.248+0000 7fc19266b640 10 monclient: monmap: 2024-03-29T10:53:18.230 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:53:18.230 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:18.231 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:53:18.248956+0000 2024-03-29T10:53:18.231 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:53:18.248956+0000 2024-03-29T10:53:18.231 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:18.231 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:18.231 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:53:18.231 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:53:18.231 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:53:18.231 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:18.233 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.249+0000 7fc19266b640 1 Processor -- start 2024-03-29T10:53:18.233 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.249+0000 7fc19266b640 1 -- start start 2024-03-29T10:53:18.233 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.249+0000 7fc19266b640 10 monclient: init 2024-03-29T10:53:18.233 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.250+0000 7fc19266b640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:18.233 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.250+0000 7fc19266b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:53:18.233 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.250+0000 7fc19266b640 10 monclient(hunting): picked mon.noname-b con 0x7fc18c0a9930 addr v1:172.21.15.112:6790/0 2024-03-29T10:53:18.233 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.250+0000 7fc19266b640 10 monclient(hunting): picked mon.noname-c con 0x7fc18c149dc0 addr v1:172.21.15.118:6789/0 2024-03-29T10:53:18.234 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.250+0000 7fc19266b640 10 monclient(hunting): picked mon.noname-a con 0x7fc18c14ff40 addr v1:172.21.15.112:6789/0 2024-03-29T10:53:18.234 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.250+0000 7fc19266b640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc18c0aae60 con 0x7fc18c14ff40 2024-03-29T10:53:18.234 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.250+0000 7fc19266b640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc18c0b4310 con 0x7fc18c149dc0 2024-03-29T10:53:18.234 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.250+0000 7fc19266b640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc18c0afdd0 con 0x7fc18c0a9930 2024-03-29T10:53:18.234 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.250+0000 7fc19266b640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:18.234 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.250+0000 7fc19266b640 10 monclient(hunting): authenticate will time out at 953.554814s 2024-03-29T10:53:18.238 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.257+0000 7fc18b7fe640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fc18c149dc0 0x7fc18c14a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:37853/0 (socket says 172.21.15.112:37853) 2024-03-29T10:53:18.239 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.257+0000 7fc18b7fe640 1 -- 172.21.15.112:0/1499961260 learned_addr learned my addr 172.21.15.112:0/1499961260 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:53:18.240 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.259+0000 7fc18affd640 1 -- 172.21.15.112:0/1499961260 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2407552127 0 0) 0x7fc18c0b4310 con 0x7fc18c149dc0 2024-03-29T10:53:18.241 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.259+0000 7fc18affd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:18.241 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.259+0000 7fc18affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:18.241 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.259+0000 7fc18affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:18.241 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.259+0000 7fc18affd640 10 monclient(hunting): my global_id is 4297 2024-03-29T10:53:18.241 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.259+0000 7fc18affd640 1 -- 172.21.15.112:0/1499961260 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc174003520 con 0x7fc18c149dc0 2024-03-29T10:53:18.248 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.260+0000 7fc18affd640 1 -- 172.21.15.112:0/1499961260 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1209045243 0 0) 0x7fc18c0afdd0 con 0x7fc18c0a9930 2024-03-29T10:53:18.248 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.260+0000 7fc18affd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:18.248 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.260+0000 7fc18affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:18.248 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.260+0000 7fc18affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:18.248 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.260+0000 7fc18affd640 10 monclient(hunting): my global_id is 4292 2024-03-29T10:53:18.248 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.260+0000 7fc18affd640 1 -- 172.21.15.112:0/1499961260 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc18c0b4310 con 0x7fc18c0a9930 2024-03-29T10:53:18.248 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.261+0000 7fc18affd640 1 -- 172.21.15.112:0/1499961260 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2274632252 0 0) 0x7fc18c0b4310 con 0x7fc18c0a9930 2024-03-29T10:53:18.249 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.261+0000 7fc18affd640 1 -- 172.21.15.112:0/1499961260 >> v1:172.21.15.118:6789/0 conn(0x7fc18c149dc0 legacy=0x7fc18c14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:18.249 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.261+0000 7fc18affd640 1 -- 172.21.15.112:0/1499961260 >> v1:172.21.15.112:6789/0 conn(0x7fc18c14ff40 legacy=0x7fc18c152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:53:18.249 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.261+0000 7fc18affd640 10 monclient: _finish_hunting 0 2024-03-29T10:53:18.249 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.261+0000 7fc18affd640 1 monclient: found mon.noname-b 2024-03-29T10:53:18.249 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.261+0000 7fc18affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:18.249 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.261+0000 7fc18affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:53:18.249 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.261+0000 7fc18affd640 1 -- 172.21.15.112:0/1499961260 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc18c14a950 con 0x7fc18c0a9930 2024-03-29T10:53:18.249 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.261+0000 7fc18affd640 10 monclient: _finish_auth 0 2024-03-29T10:53:18.249 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.261+0000 7fc18affd640 10 monclient: _check_auth_tickets 2024-03-29T10:53:18.249 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.261+0000 7fc18affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:18.249 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.261+0000 7fc18affd640 1 -- 172.21.15.112:0/1499961260 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fc180001670 con 0x7fc18c0a9930 2024-03-29T10:53:18.249 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.261+0000 7fc18affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:18.249 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.261+0000 7fc18affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:18.249 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.261+0000 7fc18affd640 10 monclient: dump: 2024-03-29T10:53:18.250 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:18.250 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:18.250 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:18.250 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:18.250 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:18.250 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:18.250 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:18.250 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:18.250 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:18.250 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:18.250 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.261+0000 7fc18affd640 1 -- 172.21.15.112:0/1499961260 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc180001890 con 0x7fc18c0a9930 2024-03-29T10:53:18.251 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.261+0000 7fc18affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:18.251 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.261+0000 7fc18affd640 1 -- 172.21.15.112:0/1499961260 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fc180001c20 con 0x7fc18c0a9930 2024-03-29T10:53:18.251 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.261+0000 7fc18affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:18.251 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.261+0000 7fc18affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:18.251 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.261+0000 7fc18affd640 10 monclient: dump: 2024-03-29T10:53:18.251 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:18.251 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:18.251 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:18.251 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:18.251 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:18.251 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:18.251 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:18.251 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:18.251 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:18.251 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:18.252 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.261+0000 7fc19266b640 5 monclient: authenticate success, global_id 4292 2024-03-29T10:53:18.252 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.261+0000 7fc19266b640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:18.252 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.261+0000 7fc19266b640 4 set_mon_vals no callback set 2024-03-29T10:53:18.252 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.262+0000 7fc19266b640 10 monclient: shutdown 2024-03-29T10:53:18.252 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.262+0000 7fc19266b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:18.252 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.262+0000 7fc19266b640 1 -- 172.21.15.112:0/1499961260 >> v1:172.21.15.112:6790/0 conn(0x7fc18c0a9930 legacy=0x7fc18c143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:18.252 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.263+0000 7fc19266b640 1 -- 172.21.15.112:0/1499961260 shutdown_connections 2024-03-29T10:53:18.252 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.263+0000 7fc19266b640 1 -- 172.21.15.112:0/1499961260 >> 172.21.15.112:0/1499961260 conn(0x7fc18c0b35e0 msgr2=0x7fc18c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:18.252 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.263+0000 7fc19266b640 1 -- 172.21.15.112:0/1499961260 shutdown_connections 2024-03-29T10:53:18.252 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.263+0000 7fc19266b640 1 -- 172.21.15.112:0/1499961260 wait complete. 2024-03-29T10:53:18.254 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.265+0000 7fc19266b640 10 monclient: build_initial_monmap 2024-03-29T10:53:18.254 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.265+0000 7fc19266b640 10 monclient: monmap: 2024-03-29T10:53:18.254 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:53:18.254 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:18.254 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:53:18.255 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:53:18.255 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:18.255 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:18.255 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:53:18.255 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:53:18.255 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:53:18.255 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:18.255 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.265+0000 7fc19266b640 1 librados: starting msgr at 2024-03-29T10:53:18.255 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.265+0000 7fc19266b640 1 librados: starting objecter 2024-03-29T10:53:18.255 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.266+0000 7fc19266b640 1 Processor -- start 2024-03-29T10:53:18.255 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.266+0000 7fc19266b640 1 -- start start 2024-03-29T10:53:18.255 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.266+0000 7fc19266b640 1 librados: setting wanted keys 2024-03-29T10:53:18.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.266+0000 7fc19266b640 1 librados: calling monclient init 2024-03-29T10:53:18.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.266+0000 7fc19266b640 10 monclient: init 2024-03-29T10:53:18.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.267+0000 7fc19266b640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:18.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.267+0000 7fc19266b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:53:18.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.270+0000 7fc19266b640 10 monclient(hunting): picked mon.noname-a con 0x7fc18c0a9930 addr v1:172.21.15.112:6789/0 2024-03-29T10:53:18.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.270+0000 7fc19266b640 10 monclient(hunting): picked mon.noname-b con 0x7fc18c149dc0 addr v1:172.21.15.118:6789/0 2024-03-29T10:53:18.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.270+0000 7fc19266b640 10 monclient(hunting): picked mon.noname-c con 0x7fc18c14ff40 addr v1:172.21.15.112:6790/0 2024-03-29T10:53:18.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.270+0000 7fc19266b640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc18c145570 con 0x7fc18c0a9930 2024-03-29T10:53:18.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.270+0000 7fc19266b640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc18c163960 con 0x7fc18c149dc0 2024-03-29T10:53:18.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.270+0000 7fc19266b640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc18c146d30 con 0x7fc18c14ff40 2024-03-29T10:53:18.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.270+0000 7fc19266b640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:18.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.270+0000 7fc19266b640 10 monclient(hunting): authenticate will time out at 953.574452s 2024-03-29T10:53:18.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.273+0000 7fc18bfff640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fc18c0a9930 0x7fc18c145a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:55225/0 (socket says 172.21.15.112:55225) 2024-03-29T10:53:18.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.273+0000 7fc18bfff640 1 -- 172.21.15.112:0/3358435709 learned_addr learned my addr 172.21.15.112:0/3358435709 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:53:18.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.275+0000 7fc188ff9640 10 client.?.objecter ms_handle_connect 0x7fc18c0a9930 2024-03-29T10:53:18.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.275+0000 7fc188ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:18.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.275+0000 7fc188ff9640 1 -- 172.21.15.112:0/3358435709 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718825053 0 0) 0x7fc18c145570 con 0x7fc18c0a9930 2024-03-29T10:53:18.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.275+0000 7fc188ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:53:18.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.275+0000 7fc188ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:18.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.275+0000 7fc188ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:18.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.275+0000 7fc188ff9640 10 monclient(hunting): my global_id is 4296 2024-03-29T10:53:18.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.275+0000 7fc188ff9640 1 -- 172.21.15.112:0/3358435709 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc178004eb0 con 0x7fc18c0a9930 2024-03-29T10:53:18.258 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.276+0000 7fc188ff9640 10 client.?.objecter ms_handle_connect 0x7fc18c149dc0 2024-03-29T10:53:18.258 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.276+0000 7fc188ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:18.259 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.276+0000 7fc188ff9640 1 -- 172.21.15.112:0/3358435709 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2149041565 0 0) 0x7fc18c163960 con 0x7fc18c149dc0 2024-03-29T10:53:18.259 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.277+0000 7fc188ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:53:18.259 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.277+0000 7fc188ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:18.259 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.277+0000 7fc188ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:18.259 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.277+0000 7fc188ff9640 10 monclient(hunting): my global_id is 4300 2024-03-29T10:53:18.259 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.277+0000 7fc188ff9640 1 -- 172.21.15.112:0/3358435709 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc18c145570 con 0x7fc18c149dc0 2024-03-29T10:53:18.259 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.277+0000 7fc188ff9640 10 client.?.objecter ms_handle_connect 0x7fc18c14ff40 2024-03-29T10:53:18.259 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.277+0000 7fc188ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:18.259 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.277+0000 7fc188ff9640 1 -- 172.21.15.112:0/3358435709 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 497705565 0 0) 0x7fc18c145570 con 0x7fc18c149dc0 2024-03-29T10:53:18.260 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.277+0000 7fc188ff9640 1 -- 172.21.15.112:0/3358435709 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc18c163960 con 0x7fc18c149dc0 2024-03-29T10:53:18.260 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.277+0000 7fc188ff9640 1 -- 172.21.15.112:0/3358435709 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fc17c001950 con 0x7fc18c149dc0 2024-03-29T10:53:18.260 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.277+0000 7fc188ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:18.260 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.277+0000 7fc188ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:18.260 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.277+0000 7fc188ff9640 10 monclient(hunting): dump: 2024-03-29T10:53:18.260 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:18.260 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:18.260 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:18.260 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:18.260 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:18.260 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:18.260 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:18.260 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:18.260 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:18.261 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:18.261 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.277+0000 7fc188ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:18.261 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.277+0000 7fc188ff9640 1 -- 172.21.15.112:0/3358435709 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442583266 0 0) 0x7fc18c146d30 con 0x7fc18c14ff40 2024-03-29T10:53:18.261 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.277+0000 7fc188ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:53:18.261 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.277+0000 7fc188ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:18.261 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.277+0000 7fc188ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:18.261 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.277+0000 7fc188ff9640 10 monclient(hunting): my global_id is 4295 2024-03-29T10:53:18.261 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.277+0000 7fc188ff9640 1 -- 172.21.15.112:0/3358435709 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc18c145570 con 0x7fc18c14ff40 2024-03-29T10:53:18.261 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.278+0000 7fc188ff9640 1 -- 172.21.15.112:0/3358435709 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 487517953 0 0) 0x7fc18c163960 con 0x7fc18c149dc0 2024-03-29T10:53:18.261 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.278+0000 7fc188ff9640 1 -- 172.21.15.112:0/3358435709 >> v1:172.21.15.112:6790/0 conn(0x7fc18c14ff40 legacy=0x7fc18c163250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:18.261 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.278+0000 7fc188ff9640 1 -- 172.21.15.112:0/3358435709 >> v1:172.21.15.112:6789/0 conn(0x7fc18c0a9930 legacy=0x7fc18c145a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:18.261 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.278+0000 7fc188ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:53:18.261 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.278+0000 7fc188ff9640 1 monclient: found mon.b 2024-03-29T10:53:18.261 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.278+0000 7fc188ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:18.262 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.278+0000 7fc188ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:53:18.262 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.278+0000 7fc188ff9640 1 -- 172.21.15.112:0/3358435709 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc18c147010 con 0x7fc18c149dc0 2024-03-29T10:53:18.262 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.278+0000 7fc188ff9640 10 monclient: _finish_auth 0 2024-03-29T10:53:18.262 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.278+0000 7fc188ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:53:18.262 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.278+0000 7fc188ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:18.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.278+0000 7fc19266b640 5 monclient: authenticate success, global_id 4300 2024-03-29T10:53:18.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.278+0000 7fc19266b640 10 monclient: _renew_subs 2024-03-29T10:53:18.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.278+0000 7fc19266b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:53:18.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.278+0000 7fc19266b640 1 -- 172.21.15.112:0/3358435709 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc18c1472f0 con 0x7fc18c149dc0 2024-03-29T10:53:18.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.278+0000 7fc19266b640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:18.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.278+0000 7fc19266b640 10 monclient: _renew_subs 2024-03-29T10:53:18.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.278+0000 7fc19266b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:53:18.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.278+0000 7fc19266b640 1 -- 172.21.15.112:0/3358435709 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc18c144510 con 0x7fc18c149dc0 2024-03-29T10:53:18.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.278+0000 7fc19266b640 1 librados: init done 2024-03-29T10:53:18.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.279+0000 7fc19266b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:18.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.279+0000 7fc19266b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:18.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.279+0000 7fc19266b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:53:18.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.279+0000 7fc19266b640 1 -- 172.21.15.112:0/3358435709 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc18c075f10 con 0x7fc18c149dc0 2024-03-29T10:53:18.267 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.283+0000 7fc188ff9640 1 -- 172.21.15.112:0/3358435709 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc17c001d60 con 0x7fc18c149dc0 2024-03-29T10:53:18.267 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.283+0000 7fc188ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:18.267 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.283+0000 7fc188ff9640 1 -- 172.21.15.112:0/3358435709 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fc17c004980 con 0x7fc18c149dc0 2024-03-29T10:53:18.267 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.283+0000 7fc188ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:18.267 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.283+0000 7fc188ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:18.267 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.283+0000 7fc188ff9640 10 monclient: dump: 2024-03-29T10:53:18.267 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:18.267 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:18.267 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:18.267 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:18.267 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:18.267 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:18.267 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:18.267 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:18.268 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:18.268 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:18.268 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.283+0000 7fc188ff9640 1 -- 172.21.15.112:0/3358435709 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fc17c01aa10 con 0x7fc18c149dc0 2024-03-29T10:53:18.268 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.284+0000 7fc18bfff640 10 monclient: get_auth_request con 0x7fc17806dde0 auth_method 0 2024-03-29T10:53:18.268 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.284+0000 7fc188ff9640 1 -- 172.21.15.112:0/3358435709 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7fc17c084a70 con 0x7fc18c149dc0 2024-03-29T10:53:18.268 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.284+0000 7fc188ff9640 10 client.4300.objecter ms_dispatch 0x7fc18c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:18.268 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.284+0000 7fc188ff9640 3 client.4300.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:18.268 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.284+0000 7fc188ff9640 3 client.4300.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:18.268 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.284+0000 7fc18a7fc640 4 set_mon_vals no callback set 2024-03-29T10:53:18.268 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.285+0000 7fc188ff9640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-29T10:53:18.268 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.285+0000 7fc188ff9640 10 client.4300.objecter ms_handle_connect 0x7fc17806dde0 2024-03-29T10:53:18.271 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.290+0000 7fc188ff9640 1 -- 172.21.15.112:0/3358435709 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc17c050200 con 0x7fc18c149dc0 2024-03-29T10:53:18.271 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.290+0000 7fc188ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:18.271 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.290+0000 7fc188ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:18.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.614+0000 7fc19266b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:18.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.614+0000 7fc19266b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:18.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.614+0000 7fc19266b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:53:18.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.614+0000 7fc19266b640 1 -- 172.21.15.112:0/3358435709 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc18c0b1860 con 0x7fc18c149dc0 2024-03-29T10:53:18.598 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.617+0000 7fc188ff9640 1 -- 172.21.15.112:0/3358435709 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7fc17c052300 con 0x7fc18c149dc0 2024-03-29T10:53:18.598 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.617+0000 7fc188ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:18.598 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.617+0000 7fc188ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:18.598 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:53:18.598 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:18.606 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.620+0000 7fc19266b640 10 librados: watch_flush enter 2024-03-29T10:53:18.606 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.620+0000 7fc19266b640 10 librados: watch_flush exit 2024-03-29T10:53:18.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.620+0000 7fc19266b640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-29T10:53:18.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.620+0000 7fc19266b640 10 client.4300.objecter successfully canceled tick 2024-03-29T10:53:18.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.623+0000 7fc19266b640 1 -- 172.21.15.112:0/3358435709 >> v1:172.21.15.118:6816/3181073408 conn(0x7fc17806dde0 legacy=0x7fc1780702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:18.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.623+0000 7fc19266b640 10 monclient: shutdown 2024-03-29T10:53:18.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.623+0000 7fc19266b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:18.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.623+0000 7fc19266b640 1 -- 172.21.15.112:0/3358435709 >> v1:172.21.15.118:6789/0 conn(0x7fc18c149dc0 legacy=0x7fc18c162b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:18.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.623+0000 7fc19266b640 1 -- 172.21.15.112:0/3358435709 shutdown_connections 2024-03-29T10:53:18.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.623+0000 7fc19266b640 1 -- 172.21.15.112:0/3358435709 >> 172.21.15.112:0/3358435709 conn(0x7fc18c0b35e0 msgr2=0x7fc18c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:18.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.624+0000 7fc19266b640 1 -- 172.21.15.112:0/3358435709 shutdown_connections 2024-03-29T10:53:18.608 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.627+0000 7fc19266b640 1 -- 172.21.15.112:0/3358435709 wait complete. 2024-03-29T10:53:18.608 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:18.627+0000 7fc19266b640 1 librados: shutdown 2024-03-29T10:53:18.626 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:25.433 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:25.586 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.604+0000 7fb6ab9c1640 10 monclient: get_monmap_and_config 2024-03-29T10:53:25.586 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.604+0000 7fb6ab9c1640 10 monclient: build_initial_monmap 2024-03-29T10:53:25.586 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.604+0000 7fb6ab9c1640 10 monclient: monmap: 2024-03-29T10:53:25.586 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:53:25.586 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:25.586 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:53:25.605081+0000 2024-03-29T10:53:25.586 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:53:25.605081+0000 2024-03-29T10:53:25.586 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:25.586 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:25.586 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:53:25.587 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:53:25.587 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:53:25.587 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:25.587 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.604+0000 7fb6ab9c1640 1 Processor -- start 2024-03-29T10:53:25.590 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.605+0000 7fb6ab9c1640 1 -- start start 2024-03-29T10:53:25.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.605+0000 7fb6ab9c1640 10 monclient: init 2024-03-29T10:53:25.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.606+0000 7fb6ab9c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:25.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.606+0000 7fb6ab9c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:25.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.606+0000 7fb6ab9c1640 10 monclient(hunting): picked mon.noname-a con 0x7fb6a40a9ac0 addr v1:172.21.15.112:6789/0 2024-03-29T10:53:25.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.606+0000 7fb6ab9c1640 10 monclient(hunting): picked mon.noname-b con 0x7fb6a413ee90 addr v1:172.21.15.112:6790/0 2024-03-29T10:53:25.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.606+0000 7fb6ab9c1640 10 monclient(hunting): picked mon.noname-c con 0x7fb6a40af900 addr v1:172.21.15.118:6789/0 2024-03-29T10:53:25.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.606+0000 7fb6ab9c1640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a40afd10 con 0x7fb6a40a9ac0 2024-03-29T10:53:25.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.606+0000 7fb6ab9c1640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a40af6f0 con 0x7fb6a40af900 2024-03-29T10:53:25.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.606+0000 7fb6ab9c1640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a40aac80 con 0x7fb6a413ee90 2024-03-29T10:53:25.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.606+0000 7fb6ab9c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:25.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.606+0000 7fb6ab9c1640 10 monclient(hunting): authenticate will time out at 960.910592s 2024-03-29T10:53:25.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.606+0000 7fb6a9f5e640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fb6a40af900 0x7fb6a4149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:44528/0 (socket says 172.21.15.112:44528) 2024-03-29T10:53:25.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.606+0000 7fb6a9f5e640 1 -- 172.21.15.112:0/3382313738 learned_addr learned my addr 172.21.15.112:0/3382313738 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:53:25.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.606+0000 7fb6a975d640 1 --1- 172.21.15.112:0/3382313738 >> v1:172.21.15.112:6789/0 conn(0x7fb6a40a9ac0 0x7fb6a40adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46446/0 (socket says 172.21.15.112:46446) 2024-03-29T10:53:25.592 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 1 -- 172.21.15.112:0/3382313738 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1242654705 0 0) 0x7fb6a40af6f0 con 0x7fb6a40af900 2024-03-29T10:53:25.592 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:25.592 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:25.592 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:25.592 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 10 monclient(hunting): my global_id is 4303 2024-03-29T10:53:25.592 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 1 -- 172.21.15.112:0/3382313738 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb690003520 con 0x7fb6a40af900 2024-03-29T10:53:25.592 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 1 -- 172.21.15.112:0/3382313738 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3073473040 0 0) 0x7fb6a40aac80 con 0x7fb6a413ee90 2024-03-29T10:53:25.592 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:25.592 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:25.592 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:25.592 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 10 monclient(hunting): my global_id is 4298 2024-03-29T10:53:25.592 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 1 -- 172.21.15.112:0/3382313738 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a40af6f0 con 0x7fb6a413ee90 2024-03-29T10:53:25.592 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 1 -- 172.21.15.112:0/3382313738 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15738466 0 0) 0x7fb6a40afd10 con 0x7fb6a40a9ac0 2024-03-29T10:53:25.592 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:25.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:25.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:25.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 10 monclient(hunting): my global_id is 4299 2024-03-29T10:53:25.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 1 -- 172.21.15.112:0/3382313738 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a40aac80 con 0x7fb6a40a9ac0 2024-03-29T10:53:25.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 1 -- 172.21.15.112:0/3382313738 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 261653536 0 0) 0x7fb690003520 con 0x7fb6a40af900 2024-03-29T10:53:25.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 1 -- 172.21.15.112:0/3382313738 >> v1:172.21.15.112:6790/0 conn(0x7fb6a413ee90 legacy=0x7fb6a413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:25.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 1 -- 172.21.15.112:0/3382313738 >> v1:172.21.15.112:6789/0 conn(0x7fb6a40a9ac0 legacy=0x7fb6a40adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:25.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:53:25.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 1 monclient: found mon.noname-c 2024-03-29T10:53:25.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:25.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T10:53:25.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 1 -- 172.21.15.112:0/3382313738 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a40aaf60 con 0x7fb6a40af900 2024-03-29T10:53:25.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 10 monclient: _finish_auth 0 2024-03-29T10:53:25.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:53:25.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:25.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 1 -- 172.21.15.112:0/3382313738 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb69c001a10 con 0x7fb6a40af900 2024-03-29T10:53:25.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:25.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:25.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb69bfff640 10 monclient: dump: 2024-03-29T10:53:25.594 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:25.594 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:25.594 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:25.594 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:25.594 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:25.594 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:25.594 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:25.594 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:25.594 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:25.595 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:25.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb6ab9c1640 5 monclient: authenticate success, global_id 4303 2024-03-29T10:53:25.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.607+0000 7fb6ab9c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:25.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.608+0000 7fb69bfff640 1 -- 172.21.15.112:0/3382313738 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb69c001bf0 con 0x7fb6a40af900 2024-03-29T10:53:25.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.608+0000 7fb69bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:25.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.608+0000 7fb6ab9c1640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:25.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.608+0000 7fb6ab9c1640 4 set_mon_vals no callback set 2024-03-29T10:53:25.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.608+0000 7fb69bfff640 1 -- 172.21.15.112:0/3382313738 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb69c002000 con 0x7fb6a40af900 2024-03-29T10:53:25.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.608+0000 7fb69bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:25.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.608+0000 7fb69bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:25.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.608+0000 7fb69bfff640 10 monclient: dump: 2024-03-29T10:53:25.595 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:25.595 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:25.595 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:25.596 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:25.596 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:25.596 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:25.596 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:25.596 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:25.596 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:25.596 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:25.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.609+0000 7fb6ab9c1640 10 monclient: shutdown 2024-03-29T10:53:25.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.609+0000 7fb6ab9c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:25.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.609+0000 7fb6ab9c1640 1 -- 172.21.15.112:0/3382313738 >> v1:172.21.15.118:6789/0 conn(0x7fb6a40af900 legacy=0x7fb6a4149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:25.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.609+0000 7fb6ab9c1640 1 -- 172.21.15.112:0/3382313738 shutdown_connections 2024-03-29T10:53:25.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.609+0000 7fb6ab9c1640 1 -- 172.21.15.112:0/3382313738 >> 172.21.15.112:0/3382313738 conn(0x7fb6a40b35e0 msgr2=0x7fb6a40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:25.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.609+0000 7fb6ab9c1640 1 -- 172.21.15.112:0/3382313738 shutdown_connections 2024-03-29T10:53:25.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.610+0000 7fb6ab9c1640 1 -- 172.21.15.112:0/3382313738 wait complete. 2024-03-29T10:53:25.604 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.610+0000 7fb6ab9c1640 10 monclient: build_initial_monmap 2024-03-29T10:53:25.604 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.610+0000 7fb6ab9c1640 10 monclient: monmap: 2024-03-29T10:53:25.604 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:53:25.604 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:25.604 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:53:25.604 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:53:25.604 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:25.604 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:25.604 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:53:25.604 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:53:25.604 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:53:25.604 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:25.604 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.611+0000 7fb6ab9c1640 1 librados: starting msgr at 2024-03-29T10:53:25.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.611+0000 7fb6ab9c1640 1 librados: starting objecter 2024-03-29T10:53:25.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.611+0000 7fb6ab9c1640 1 Processor -- start 2024-03-29T10:53:25.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.611+0000 7fb6ab9c1640 1 -- start start 2024-03-29T10:53:25.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.611+0000 7fb6ab9c1640 1 librados: setting wanted keys 2024-03-29T10:53:25.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.611+0000 7fb6ab9c1640 1 librados: calling monclient init 2024-03-29T10:53:25.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.611+0000 7fb6ab9c1640 10 monclient: init 2024-03-29T10:53:25.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.613+0000 7fb6ab9c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:25.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.614+0000 7fb6ab9c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:25.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.614+0000 7fb6ab9c1640 10 monclient(hunting): picked mon.noname-a con 0x7fb6a40a9ac0 addr v1:172.21.15.112:6789/0 2024-03-29T10:53:25.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.614+0000 7fb6ab9c1640 10 monclient(hunting): picked mon.noname-c con 0x7fb6a40af900 addr v1:172.21.15.112:6790/0 2024-03-29T10:53:25.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.614+0000 7fb6ab9c1640 10 monclient(hunting): picked mon.noname-b con 0x7fb6a413ee90 addr v1:172.21.15.118:6789/0 2024-03-29T10:53:25.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.614+0000 7fb6ab9c1640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a4137190 con 0x7fb6a40a9ac0 2024-03-29T10:53:25.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.614+0000 7fb6a9f5e640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fb6a413ee90 0x7fb6a4136a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:44760/0 (socket says 172.21.15.112:44760) 2024-03-29T10:53:25.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.614+0000 7fb6a9f5e640 1 -- 172.21.15.112:0/927922040 learned_addr learned my addr 172.21.15.112:0/927922040 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:53:25.606 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.614+0000 7fb699ffb640 10 client.?.objecter ms_handle_connect 0x7fb6a413ee90 2024-03-29T10:53:25.606 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.614+0000 7fb699ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:25.606 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.614+0000 7fb699ffb640 10 client.?.objecter ms_handle_connect 0x7fb6a40af900 2024-03-29T10:53:25.606 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.614+0000 7fb699ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:25.606 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.615+0000 7fb6ab9c1640 1 -- 172.21.15.112:0/927922040 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a406ed20 con 0x7fb6a413ee90 2024-03-29T10:53:25.606 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.615+0000 7fb6ab9c1640 1 -- 172.21.15.112:0/927922040 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a406f000 con 0x7fb6a40af900 2024-03-29T10:53:25.606 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.615+0000 7fb6ab9c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:25.606 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.615+0000 7fb6ab9c1640 10 monclient(hunting): authenticate will time out at 960.919324s 2024-03-29T10:53:25.606 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.615+0000 7fb699ffb640 10 client.?.objecter ms_handle_connect 0x7fb6a40a9ac0 2024-03-29T10:53:25.606 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.615+0000 7fb699ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:25.606 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.615+0000 7fb699ffb640 1 -- 172.21.15.112:0/927922040 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582063215 0 0) 0x7fb6a406f000 con 0x7fb6a40af900 2024-03-29T10:53:25.606 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.615+0000 7fb699ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:53:25.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.615+0000 7fb699ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:25.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.615+0000 7fb699ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:25.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.615+0000 7fb699ffb640 10 monclient(hunting): my global_id is 4301 2024-03-29T10:53:25.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.615+0000 7fb699ffb640 1 -- 172.21.15.112:0/927922040 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb688004eb0 con 0x7fb6a40af900 2024-03-29T10:53:25.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.616+0000 7fb699ffb640 1 -- 172.21.15.112:0/927922040 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 240580861 0 0) 0x7fb6a406ed20 con 0x7fb6a413ee90 2024-03-29T10:53:25.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.616+0000 7fb699ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:53:25.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.616+0000 7fb699ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:25.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.616+0000 7fb699ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:25.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.616+0000 7fb699ffb640 10 monclient(hunting): my global_id is 4306 2024-03-29T10:53:25.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.616+0000 7fb699ffb640 1 -- 172.21.15.112:0/927922040 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a406f000 con 0x7fb6a413ee90 2024-03-29T10:53:25.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.616+0000 7fb699ffb640 1 -- 172.21.15.112:0/927922040 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 882269107 0 0) 0x7fb6a4137190 con 0x7fb6a40a9ac0 2024-03-29T10:53:25.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.616+0000 7fb699ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:53:25.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.616+0000 7fb699ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:25.608 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.616+0000 7fb699ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:25.608 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.616+0000 7fb699ffb640 10 monclient(hunting): my global_id is 4302 2024-03-29T10:53:25.608 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.616+0000 7fb699ffb640 1 -- 172.21.15.112:0/927922040 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a406ed20 con 0x7fb6a40a9ac0 2024-03-29T10:53:25.608 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb699ffb640 1 -- 172.21.15.112:0/927922040 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2116437893 0 0) 0x7fb688004eb0 con 0x7fb6a40af900 2024-03-29T10:53:25.608 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb699ffb640 1 -- 172.21.15.112:0/927922040 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6a4137190 con 0x7fb6a40af900 2024-03-29T10:53:25.608 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb699ffb640 1 -- 172.21.15.112:0/927922040 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb694001810 con 0x7fb6a40af900 2024-03-29T10:53:25.608 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb699ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:25.608 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb699ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:25.608 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb699ffb640 10 monclient(hunting): dump: 2024-03-29T10:53:25.608 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:25.608 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:25.608 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:25.608 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:25.608 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:25.609 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:25.609 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:25.609 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:25.609 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:25.609 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:25.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb699ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:25.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb699ffb640 1 -- 172.21.15.112:0/927922040 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1591291556 0 0) 0x7fb6a406f000 con 0x7fb6a413ee90 2024-03-29T10:53:25.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb699ffb640 1 -- 172.21.15.112:0/927922040 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb688004eb0 con 0x7fb6a413ee90 2024-03-29T10:53:25.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb699ffb640 1 -- 172.21.15.112:0/927922040 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb69c001f00 con 0x7fb6a413ee90 2024-03-29T10:53:25.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb699ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:25.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb699ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:25.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb699ffb640 10 monclient(hunting): dump: 2024-03-29T10:53:25.609 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:25.609 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:25.610 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:25.610 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:25.610 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:25.610 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:25.610 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:25.610 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:25.610 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:25.610 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:25.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb699ffb640 1 -- 172.21.15.112:0/927922040 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 242172634 0 0) 0x7fb688004eb0 con 0x7fb6a413ee90 2024-03-29T10:53:25.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb699ffb640 1 -- 172.21.15.112:0/927922040 >> v1:172.21.15.112:6790/0 conn(0x7fb6a40af900 legacy=0x7fb6a4136370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:25.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb699ffb640 1 -- 172.21.15.112:0/927922040 >> v1:172.21.15.112:6789/0 conn(0x7fb6a40a9ac0 legacy=0x7fb6a41d32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:25.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb699ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:53:25.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb699ffb640 1 monclient: found mon.b 2024-03-29T10:53:25.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb699ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:25.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb699ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:53:25.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb699ffb640 1 -- 172.21.15.112:0/927922040 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a4140eb0 con 0x7fb6a413ee90 2024-03-29T10:53:25.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb699ffb640 10 monclient: _finish_auth 0 2024-03-29T10:53:25.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb699ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:53:25.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb699ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:25.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb6ab9c1640 5 monclient: authenticate success, global_id 4306 2024-03-29T10:53:25.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb6ab9c1640 10 monclient: _renew_subs 2024-03-29T10:53:25.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb6ab9c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:53:25.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.617+0000 7fb6ab9c1640 1 -- 172.21.15.112:0/927922040 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a4141630 con 0x7fb6a413ee90 2024-03-29T10:53:25.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.618+0000 7fb6ab9c1640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:25.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.618+0000 7fb6ab9c1640 10 monclient: _renew_subs 2024-03-29T10:53:25.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.618+0000 7fb6ab9c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:53:25.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.618+0000 7fb6ab9c1640 1 -- 172.21.15.112:0/927922040 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a4141110 con 0x7fb6a413ee90 2024-03-29T10:53:25.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.618+0000 7fb6ab9c1640 1 librados: init done 2024-03-29T10:53:25.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.618+0000 7fb699ffb640 1 -- 172.21.15.112:0/927922040 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb69c002060 con 0x7fb6a413ee90 2024-03-29T10:53:25.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.618+0000 7fb699ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:25.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.618+0000 7fb699ffb640 1 -- 172.21.15.112:0/927922040 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb69c005110 con 0x7fb6a413ee90 2024-03-29T10:53:25.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.618+0000 7fb699ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:25.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.618+0000 7fb699ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:25.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.618+0000 7fb699ffb640 10 monclient: dump: 2024-03-29T10:53:25.612 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:25.612 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:25.612 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:25.612 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:25.612 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:25.612 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:25.612 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:25.612 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:25.612 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:25.613 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:25.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.618+0000 7fb69b7fe640 4 set_mon_vals no callback set 2024-03-29T10:53:25.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.620+0000 7fb699ffb640 1 -- 172.21.15.112:0/927922040 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fb69c01b0e0 con 0x7fb6a413ee90 2024-03-29T10:53:25.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.621+0000 7fb699ffb640 1 -- 172.21.15.112:0/927922040 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7fb69c085090 con 0x7fb6a413ee90 2024-03-29T10:53:25.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.621+0000 7fb699ffb640 10 client.4306.objecter ms_dispatch 0x7fb6a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:25.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.621+0000 7fb699ffb640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:25.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.621+0000 7fb699ffb640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:25.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.621+0000 7fb699ffb640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-29T10:53:25.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.622+0000 7fb6a975d640 10 monclient: get_auth_request con 0x7fb6880723c0 auth_method 0 2024-03-29T10:53:25.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.622+0000 7fb6ab9c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:25.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.622+0000 7fb6ab9c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:25.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.622+0000 7fb6ab9c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:53:25.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.622+0000 7fb6ab9c1640 1 -- 172.21.15.112:0/927922040 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a413bcd0 con 0x7fb6a413ee90 2024-03-29T10:53:25.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.622+0000 7fb699ffb640 10 client.4306.objecter ms_handle_connect 0x7fb6880723c0 2024-03-29T10:53:25.614 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.629+0000 7fb699ffb640 1 -- 172.21.15.112:0/927922040 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb69c0507a0 con 0x7fb6a413ee90 2024-03-29T10:53:25.614 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.629+0000 7fb699ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:25.614 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:25.629+0000 7fb699ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:26.007 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:26.024+0000 7fb6ab9c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:26.227 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:26.024+0000 7fb6ab9c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:26.227 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:26.024+0000 7fb6ab9c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:53:26.227 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:26.024+0000 7fb6ab9c1640 1 -- 172.21.15.112:0/927922040 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6a413f3b0 con 0x7fb6a413ee90 2024-03-29T10:53:26.228 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:26.025+0000 7fb699ffb640 1 -- 172.21.15.112:0/927922040 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7fb69c0528a0 con 0x7fb6a413ee90 2024-03-29T10:53:26.228 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:26.025+0000 7fb699ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:26.229 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:26.025+0000 7fb699ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:26.229 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:53:26.229 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:26.230 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:26.029+0000 7fb6ab9c1640 10 librados: watch_flush enter 2024-03-29T10:53:26.230 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:26.029+0000 7fb6ab9c1640 10 librados: watch_flush exit 2024-03-29T10:53:26.230 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:26.029+0000 7fb6ab9c1640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-29T10:53:26.231 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:26.029+0000 7fb6ab9c1640 10 client.4306.objecter successfully canceled tick 2024-03-29T10:53:26.231 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:26.029+0000 7fb6ab9c1640 1 -- 172.21.15.112:0/927922040 >> v1:172.21.15.118:6816/3181073408 conn(0x7fb6880723c0 legacy=0x7fb688074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:26.231 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:26.029+0000 7fb6ab9c1640 10 monclient: shutdown 2024-03-29T10:53:26.231 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:26.029+0000 7fb6ab9c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:26.231 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:26.029+0000 7fb6ab9c1640 1 -- 172.21.15.112:0/927922040 >> v1:172.21.15.118:6789/0 conn(0x7fb6a413ee90 legacy=0x7fb6a4136a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:26.231 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:26.029+0000 7fb6ab9c1640 1 -- 172.21.15.112:0/927922040 shutdown_connections 2024-03-29T10:53:26.231 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:26.029+0000 7fb6ab9c1640 1 -- 172.21.15.112:0/927922040 >> 172.21.15.112:0/927922040 conn(0x7fb6a40b35e0 msgr2=0x7fb6a4137740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:26.231 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:26.029+0000 7fb6ab9c1640 1 -- 172.21.15.112:0/927922040 shutdown_connections 2024-03-29T10:53:26.231 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:26.029+0000 7fb6ab9c1640 1 -- 172.21.15.112:0/927922040 wait complete. 2024-03-29T10:53:26.231 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:26.029+0000 7fb6ab9c1640 1 librados: shutdown 2024-03-29T10:53:26.231 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:33.038 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:33.184 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.202+0000 7f3541efb640 10 monclient: get_monmap_and_config 2024-03-29T10:53:33.184 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.202+0000 7f3541efb640 10 monclient: build_initial_monmap 2024-03-29T10:53:33.184 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.202+0000 7f3541efb640 10 monclient: monmap: 2024-03-29T10:53:33.184 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:53:33.184 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:33.184 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:53:33.203762+0000 2024-03-29T10:53:33.184 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:53:33.203762+0000 2024-03-29T10:53:33.184 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:33.184 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:33.184 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:53:33.184 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:53:33.184 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:53:33.185 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:33.185 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.204+0000 7f3541efb640 1 Processor -- start 2024-03-29T10:53:33.185 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.204+0000 7f3541efb640 1 -- start start 2024-03-29T10:53:33.185 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.204+0000 7f3541efb640 10 monclient: init 2024-03-29T10:53:33.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.205+0000 7f3541efb640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:33.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.205+0000 7f3541efb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:33.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.205+0000 7f3541efb640 10 monclient(hunting): picked mon.noname-a con 0x7f353c071de0 addr v1:172.21.15.112:6789/0 2024-03-29T10:53:33.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.205+0000 7f353b7fe640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f353c071de0 0x7f353c0aed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36818/0 (socket says 172.21.15.112:36818) 2024-03-29T10:53:33.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.205+0000 7f353b7fe640 1 -- 172.21.15.112:0/1427318562 learned_addr learned my addr 172.21.15.112:0/1427318562 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:53:33.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.206+0000 7f3541efb640 10 monclient(hunting): picked mon.noname-b con 0x7f353c0721c0 addr v1:172.21.15.112:6790/0 2024-03-29T10:53:33.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.206+0000 7f3541efb640 10 monclient(hunting): picked mon.noname-c con 0x7f353c145ae0 addr v1:172.21.15.118:6789/0 2024-03-29T10:53:33.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.206+0000 7f3541efb640 1 -- 172.21.15.112:0/1427318562 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f353c145650 con 0x7f353c071de0 2024-03-29T10:53:33.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.206+0000 7f3541efb640 1 -- 172.21.15.112:0/1427318562 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f353c0ae260 con 0x7f353c145ae0 2024-03-29T10:53:33.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.206+0000 7f3541efb640 1 -- 172.21.15.112:0/1427318562 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f353c1420d0 con 0x7f353c0721c0 2024-03-29T10:53:33.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.206+0000 7f3541efb640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:33.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.206+0000 7f3541efb640 10 monclient(hunting): authenticate will time out at 968.510392s 2024-03-29T10:53:33.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.206+0000 7f353a7fc640 1 -- 172.21.15.112:0/1427318562 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632141273 0 0) 0x7f353c145650 con 0x7f353c071de0 2024-03-29T10:53:33.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.206+0000 7f353a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:33.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.206+0000 7f353a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:33.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.206+0000 7f353a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:33.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.206+0000 7f353a7fc640 10 monclient(hunting): my global_id is 4305 2024-03-29T10:53:33.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.206+0000 7f353a7fc640 1 -- 172.21.15.112:0/1427318562 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3524003520 con 0x7f353c071de0 2024-03-29T10:53:33.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f353a7fc640 1 -- 172.21.15.112:0/1427318562 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2739598083 0 0) 0x7f3524003520 con 0x7f353c071de0 2024-03-29T10:53:33.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f353a7fc640 1 -- 172.21.15.112:0/1427318562 >> v1:172.21.15.112:6790/0 conn(0x7f353c0721c0 legacy=0x7f353c13f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:33.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f353a7fc640 1 -- 172.21.15.112:0/1427318562 >> v1:172.21.15.118:6789/0 conn(0x7f353c145ae0 legacy=0x7f353c145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:33.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f353a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:53:33.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f353a7fc640 1 monclient: found mon.noname-a 2024-03-29T10:53:33.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f353a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:33.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f353a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T10:53:33.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f353a7fc640 1 -- 172.21.15.112:0/1427318562 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f353c0b0590 con 0x7f353c071de0 2024-03-29T10:53:33.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f353a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:53:33.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f353a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:53:33.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f353a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:33.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f3541efb640 5 monclient: authenticate success, global_id 4305 2024-03-29T10:53:33.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f3541efb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:33.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f353a7fc640 1 -- 172.21.15.112:0/1427318562 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f352c001690 con 0x7f353c071de0 2024-03-29T10:53:33.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f353a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:33.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f353a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:33.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f353a7fc640 10 monclient: dump: 2024-03-29T10:53:33.190 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:33.190 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:33.190 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:33.190 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:33.190 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:33.190 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:33.190 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:33.190 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:33.190 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:33.190 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:33.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f3541efb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:33.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f353a7fc640 1 -- 172.21.15.112:0/1427318562 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f352c0018b0 con 0x7f353c071de0 2024-03-29T10:53:33.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f353a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:33.192 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f353a7fc640 1 -- 172.21.15.112:0/1427318562 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f352c001bc0 con 0x7f353c071de0 2024-03-29T10:53:33.192 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f353a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:33.192 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f353a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:33.192 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f353a7fc640 10 monclient: dump: 2024-03-29T10:53:33.192 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:33.192 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:33.192 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:33.192 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:33.192 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:33.192 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:33.192 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:33.192 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:33.192 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:33.192 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:33.192 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f3541efb640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:33.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.207+0000 7f3541efb640 4 set_mon_vals no callback set 2024-03-29T10:53:33.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.208+0000 7f3541efb640 10 monclient: shutdown 2024-03-29T10:53:33.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.208+0000 7f3541efb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:33.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.208+0000 7f3541efb640 1 -- 172.21.15.112:0/1427318562 >> v1:172.21.15.112:6789/0 conn(0x7f353c071de0 legacy=0x7f353c0aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:33.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.209+0000 7f3541efb640 1 -- 172.21.15.112:0/1427318562 shutdown_connections 2024-03-29T10:53:33.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.209+0000 7f3541efb640 1 -- 172.21.15.112:0/1427318562 >> 172.21.15.112:0/1427318562 conn(0x7f353c06b660 msgr2=0x7f353c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:33.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.209+0000 7f3541efb640 1 -- 172.21.15.112:0/1427318562 shutdown_connections 2024-03-29T10:53:33.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.209+0000 7f3541efb640 1 -- 172.21.15.112:0/1427318562 wait complete. 2024-03-29T10:53:33.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.210+0000 7f3541efb640 10 monclient: build_initial_monmap 2024-03-29T10:53:33.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.210+0000 7f3541efb640 10 monclient: monmap: 2024-03-29T10:53:33.193 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:53:33.193 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:33.193 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:53:33.193 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:53:33.193 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:33.194 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:33.194 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:53:33.194 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:53:33.194 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:53:33.194 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:33.200 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.211+0000 7f3541efb640 1 librados: starting msgr at 2024-03-29T10:53:33.200 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.211+0000 7f3541efb640 1 librados: starting objecter 2024-03-29T10:53:33.201 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.211+0000 7f3541efb640 1 Processor -- start 2024-03-29T10:53:33.201 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.211+0000 7f3541efb640 1 -- start start 2024-03-29T10:53:33.201 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.211+0000 7f3541efb640 1 librados: setting wanted keys 2024-03-29T10:53:33.201 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.211+0000 7f3541efb640 1 librados: calling monclient init 2024-03-29T10:53:33.201 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.211+0000 7f3541efb640 10 monclient: init 2024-03-29T10:53:33.201 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.212+0000 7f3541efb640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:33.201 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.212+0000 7f3541efb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:33.201 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.212+0000 7f3541efb640 10 monclient(hunting): picked mon.noname-a con 0x7f353c071de0 addr v1:172.21.15.112:6789/0 2024-03-29T10:53:33.201 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.212+0000 7f3541efb640 10 monclient(hunting): picked mon.noname-c con 0x7f353c145ae0 addr v1:172.21.15.112:6790/0 2024-03-29T10:53:33.201 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.212+0000 7f3541efb640 10 monclient(hunting): picked mon.noname-b con 0x7f353c144ba0 addr v1:172.21.15.118:6789/0 2024-03-29T10:53:33.201 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.212+0000 7f3541efb640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f353c06f1f0 con 0x7f353c071de0 2024-03-29T10:53:33.201 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.212+0000 7f3541efb640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f353c06f460 con 0x7f353c144ba0 2024-03-29T10:53:33.201 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.212+0000 7f3541efb640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f353c06f740 con 0x7f353c145ae0 2024-03-29T10:53:33.201 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.212+0000 7f3541efb640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:33.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.212+0000 7f3541efb640 10 monclient(hunting): authenticate will time out at 968.517135s 2024-03-29T10:53:33.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.213+0000 7f353bfff640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f353c144ba0 0x7f353c1e5f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:56248/0 (socket says 172.21.15.112:56248) 2024-03-29T10:53:33.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.213+0000 7f353bfff640 1 -- 172.21.15.112:0/245916109 learned_addr learned my addr 172.21.15.112:0/245916109 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:53:33.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.213+0000 7f3523fff640 10 client.?.objecter ms_handle_connect 0x7f353c071de0 2024-03-29T10:53:33.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.213+0000 7f3523fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:33.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.213+0000 7f3523fff640 10 client.?.objecter ms_handle_connect 0x7f353c144ba0 2024-03-29T10:53:33.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.213+0000 7f3523fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:33.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.213+0000 7f3523fff640 10 client.?.objecter ms_handle_connect 0x7f353c145ae0 2024-03-29T10:53:33.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.213+0000 7f3523fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:33.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.213+0000 7f3523fff640 1 -- 172.21.15.112:0/245916109 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112621013 0 0) 0x7f353c06f740 con 0x7f353c145ae0 2024-03-29T10:53:33.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.213+0000 7f3523fff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:33.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.213+0000 7f3523fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:33.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.213+0000 7f3523fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:33.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.213+0000 7f3523fff640 10 monclient(hunting): my global_id is 4304 2024-03-29T10:53:33.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.213+0000 7f3523fff640 1 -- 172.21.15.112:0/245916109 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3528004eb0 con 0x7f353c145ae0 2024-03-29T10:53:33.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.213+0000 7f3523fff640 1 -- 172.21.15.112:0/245916109 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178124217 0 0) 0x7f353c06f1f0 con 0x7f353c071de0 2024-03-29T10:53:33.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.213+0000 7f3523fff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:33.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.213+0000 7f3523fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:33.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.213+0000 7f3523fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:33.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.213+0000 7f3523fff640 10 monclient(hunting): my global_id is 4308 2024-03-29T10:53:33.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.213+0000 7f3523fff640 1 -- 172.21.15.112:0/245916109 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f353c06f740 con 0x7f353c071de0 2024-03-29T10:53:33.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.214+0000 7f3523fff640 1 -- 172.21.15.112:0/245916109 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3298493842 0 0) 0x7f3528004eb0 con 0x7f353c145ae0 2024-03-29T10:53:33.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.214+0000 7f3523fff640 1 -- 172.21.15.112:0/245916109 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f353c06f1f0 con 0x7f353c145ae0 2024-03-29T10:53:33.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.214+0000 7f3523fff640 1 -- 172.21.15.112:0/245916109 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f3530001930 con 0x7f353c145ae0 2024-03-29T10:53:33.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.214+0000 7f3523fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:33.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.214+0000 7f3523fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:33.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.214+0000 7f3523fff640 10 monclient(hunting): dump: 2024-03-29T10:53:33.203 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:33.203 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:33.204 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:33.204 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:33.204 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:33.204 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:33.204 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:33.204 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:33.204 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:33.204 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:33.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.214+0000 7f3523fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:33.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.214+0000 7f3523fff640 1 -- 172.21.15.112:0/245916109 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 959701389 0 0) 0x7f353c06f740 con 0x7f353c071de0 2024-03-29T10:53:33.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.214+0000 7f3523fff640 1 -- 172.21.15.112:0/245916109 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35280069a0 con 0x7f353c071de0 2024-03-29T10:53:33.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.214+0000 7f3523fff640 1 -- 172.21.15.112:0/245916109 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f352c001550 con 0x7f353c071de0 2024-03-29T10:53:33.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.214+0000 7f3523fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:33.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.214+0000 7f3523fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:33.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.214+0000 7f3523fff640 10 monclient(hunting): dump: 2024-03-29T10:53:33.205 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:33.205 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:33.205 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:33.205 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:33.205 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:33.205 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:33.205 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:33.205 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:33.205 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:33.205 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:33.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.215+0000 7f3523fff640 1 -- 172.21.15.112:0/245916109 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2486415331 0 0) 0x7f353c06f1f0 con 0x7f353c145ae0 2024-03-29T10:53:33.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.215+0000 7f3523fff640 1 -- 172.21.15.112:0/245916109 >> v1:172.21.15.118:6789/0 conn(0x7f353c144ba0 legacy=0x7f353c1e5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:33.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.215+0000 7f3523fff640 1 -- 172.21.15.112:0/245916109 >> v1:172.21.15.112:6789/0 conn(0x7f353c071de0 legacy=0x7f353c06dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:33.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.215+0000 7f3523fff640 10 monclient: _finish_hunting 0 2024-03-29T10:53:33.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.215+0000 7f3523fff640 1 monclient: found mon.c 2024-03-29T10:53:33.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.215+0000 7f3523fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:33.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.215+0000 7f3523fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:53:33.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.215+0000 7f3523fff640 1 -- 172.21.15.112:0/245916109 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f353c06fa20 con 0x7f353c145ae0 2024-03-29T10:53:33.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.215+0000 7f3523fff640 10 monclient: _finish_auth 0 2024-03-29T10:53:33.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.215+0000 7f3523fff640 10 monclient: _check_auth_tickets 2024-03-29T10:53:33.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.215+0000 7f3523fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:33.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.215+0000 7f3541efb640 5 monclient: authenticate success, global_id 4304 2024-03-29T10:53:33.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.215+0000 7f3541efb640 10 monclient: _renew_subs 2024-03-29T10:53:33.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.215+0000 7f3541efb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:53:33.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.215+0000 7f3541efb640 1 -- 172.21.15.112:0/245916109 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f353c0ad230 con 0x7f353c145ae0 2024-03-29T10:53:33.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.215+0000 7f3541efb640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:33.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.215+0000 7f3541efb640 10 monclient: _renew_subs 2024-03-29T10:53:33.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.215+0000 7f3541efb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:53:33.207 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.215+0000 7f3541efb640 1 -- 172.21.15.112:0/245916109 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f353c1e7640 con 0x7f353c145ae0 2024-03-29T10:53:33.207 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.215+0000 7f3541efb640 1 librados: init done 2024-03-29T10:53:33.207 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.216+0000 7f3523fff640 1 -- 172.21.15.112:0/245916109 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35300035f0 con 0x7f353c145ae0 2024-03-29T10:53:33.207 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.216+0000 7f3523fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:33.207 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.216+0000 7f3523fff640 1 -- 172.21.15.112:0/245916109 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f3530001930 con 0x7f353c145ae0 2024-03-29T10:53:33.207 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.216+0000 7f3523fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:33.207 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.216+0000 7f3523fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:33.207 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.216+0000 7f3523fff640 10 monclient: dump: 2024-03-29T10:53:33.207 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:33.207 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:33.207 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:33.207 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:33.207 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:33.207 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:33.207 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:33.207 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:33.208 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:33.208 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:33.208 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.216+0000 7f3539ffb640 4 set_mon_vals no callback set 2024-03-29T10:53:33.208 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.216+0000 7f3541efb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:33.208 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.217+0000 7f3541efb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:33.208 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.217+0000 7f3541efb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:53:33.208 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.217+0000 7f3541efb640 1 -- 172.21.15.112:0/245916109 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f351c005080 con 0x7f353c145ae0 2024-03-29T10:53:33.208 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.218+0000 7f3523fff640 1 -- 172.21.15.112:0/245916109 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f3530003820 con 0x7f353c145ae0 2024-03-29T10:53:33.208 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.218+0000 7f3523fff640 1 -- 172.21.15.112:0/245916109 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f3530083350 con 0x7f353c145ae0 2024-03-29T10:53:33.208 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.219+0000 7f3523fff640 10 client.4304.objecter ms_dispatch 0x7f353c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:33.208 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.219+0000 7f3523fff640 3 client.4304.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:33.208 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.219+0000 7f3523fff640 3 client.4304.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:33.208 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.219+0000 7f3523fff640 20 client.4304.objecter dump_active .. 0 homeless 2024-03-29T10:53:33.208 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.219+0000 7f353b7fe640 10 monclient: get_auth_request con 0x7f352806e3f0 auth_method 0 2024-03-29T10:53:33.208 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.219+0000 7f3523fff640 10 client.4304.objecter ms_handle_connect 0x7f352806e3f0 2024-03-29T10:53:33.209 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.228+0000 7f3523fff640 1 -- 172.21.15.112:0/245916109 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f353004ed70 con 0x7f353c145ae0 2024-03-29T10:53:33.209 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.228+0000 7f3523fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:33.209 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.228+0000 7f3523fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:33.579 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.597+0000 7f3541efb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:33.579 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.597+0000 7f3541efb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:33.579 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.597+0000 7f3541efb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:53:33.579 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.597+0000 7f3541efb640 1 -- 172.21.15.112:0/245916109 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f351c005330 con 0x7f353c145ae0 2024-03-29T10:53:33.579 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.598+0000 7f3523fff640 1 -- 172.21.15.112:0/245916109 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f353004ed70 con 0x7f353c145ae0 2024-03-29T10:53:33.579 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.598+0000 7f3523fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:33.580 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.598+0000 7f3523fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:33.580 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:53:33.580 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:33.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.602+0000 7f3541efb640 10 librados: watch_flush enter 2024-03-29T10:53:33.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.602+0000 7f3541efb640 10 librados: watch_flush exit 2024-03-29T10:53:33.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.602+0000 7f3541efb640 20 client.4304.objecter shutdown clearing up homeless session... 2024-03-29T10:53:33.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.602+0000 7f3541efb640 10 client.4304.objecter successfully canceled tick 2024-03-29T10:53:33.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.602+0000 7f3541efb640 1 -- 172.21.15.112:0/245916109 >> v1:172.21.15.118:6816/3181073408 conn(0x7f352806e3f0 legacy=0x7f35280708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:33.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.602+0000 7f3541efb640 10 monclient: shutdown 2024-03-29T10:53:33.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.602+0000 7f3541efb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:33.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.602+0000 7f3541efb640 1 -- 172.21.15.112:0/245916109 >> v1:172.21.15.112:6790/0 conn(0x7f353c145ae0 legacy=0x7f353c06e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:33.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.602+0000 7f3541efb640 1 -- 172.21.15.112:0/245916109 shutdown_connections 2024-03-29T10:53:33.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.602+0000 7f3541efb640 1 -- 172.21.15.112:0/245916109 >> 172.21.15.112:0/245916109 conn(0x7f353c06b660 msgr2=0x7f353c0b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:33.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.603+0000 7f3541efb640 1 -- 172.21.15.112:0/245916109 shutdown_connections 2024-03-29T10:53:33.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.603+0000 7f3541efb640 1 -- 172.21.15.112:0/245916109 wait complete. 2024-03-29T10:53:33.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:33.603+0000 7f3541efb640 1 librados: shutdown 2024-03-29T10:53:33.599 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:40.405 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:40.538 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.557+0000 7fe578a2a640 10 monclient: get_monmap_and_config 2024-03-29T10:53:40.538 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.557+0000 7fe578a2a640 10 monclient: build_initial_monmap 2024-03-29T10:53:40.539 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.557+0000 7fe578a2a640 10 monclient: monmap: 2024-03-29T10:53:40.539 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:53:40.539 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:40.539 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:53:40.558372+0000 2024-03-29T10:53:40.539 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:53:40.558372+0000 2024-03-29T10:53:40.539 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:40.539 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:40.539 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:53:40.539 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:53:40.539 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:53:40.539 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:40.540 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.558+0000 7fe578a2a640 1 Processor -- start 2024-03-29T10:53:40.540 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.558+0000 7fe578a2a640 1 -- start start 2024-03-29T10:53:40.540 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.558+0000 7fe578a2a640 10 monclient: init 2024-03-29T10:53:40.540 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.559+0000 7fe578a2a640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:40.540 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.559+0000 7fe578a2a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:53:40.540 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.559+0000 7fe578a2a640 10 monclient(hunting): picked mon.noname-b con 0x7fe57406c1e0 addr v1:172.21.15.112:6790/0 2024-03-29T10:53:40.540 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.559+0000 7fe578a2a640 10 monclient(hunting): picked mon.noname-a con 0x7fe57406c5c0 addr v1:172.21.15.112:6789/0 2024-03-29T10:53:40.540 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.559+0000 7fe578a2a640 10 monclient(hunting): picked mon.noname-c con 0x7fe5740ae590 addr v1:172.21.15.118:6789/0 2024-03-29T10:53:40.541 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.559+0000 7fe578a2a640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe57413fc80 con 0x7fe57406c5c0 2024-03-29T10:53:40.541 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.559+0000 7fe578a2a640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe57406d5c0 con 0x7fe5740ae590 2024-03-29T10:53:40.541 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.559+0000 7fe578a2a640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe574146430 con 0x7fe57406c1e0 2024-03-29T10:53:40.541 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.559+0000 7fe578a2a640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:40.541 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.559+0000 7fe578a2a640 10 monclient(hunting): authenticate will time out at 975.863654s 2024-03-29T10:53:40.541 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.559+0000 7fe572d9d640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fe5740ae590 0x7fe57413f570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:54460/0 (socket says 172.21.15.112:54460) 2024-03-29T10:53:40.542 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.559+0000 7fe572d9d640 1 -- 172.21.15.112:0/84652003 learned_addr learned my addr 172.21.15.112:0/84652003 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:53:40.542 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.560+0000 7fe57159a640 1 -- 172.21.15.112:0/84652003 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787679769 0 0) 0x7fe574146430 con 0x7fe57406c1e0 2024-03-29T10:53:40.542 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.560+0000 7fe57159a640 10 monclient(hunting): _negotiate 2024-03-29T10:53:40.542 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.560+0000 7fe57159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:40.542 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.560+0000 7fe57159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:40.542 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.560+0000 7fe57159a640 10 monclient(hunting): my global_id is 4307 2024-03-29T10:53:40.542 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.560+0000 7fe57159a640 1 -- 172.21.15.112:0/84652003 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe55c003520 con 0x7fe57406c1e0 2024-03-29T10:53:40.542 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.560+0000 7fe57159a640 1 -- 172.21.15.112:0/84652003 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2873014617 0 0) 0x7fe55c003520 con 0x7fe57406c1e0 2024-03-29T10:53:40.542 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.560+0000 7fe57159a640 1 -- 172.21.15.112:0/84652003 >> v1:172.21.15.118:6789/0 conn(0x7fe5740ae590 legacy=0x7fe57413f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:40.542 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.560+0000 7fe57159a640 1 -- 172.21.15.112:0/84652003 >> v1:172.21.15.112:6789/0 conn(0x7fe57406c5c0 legacy=0x7fe57413ee60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:40.544 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.563+0000 7fe57159a640 10 monclient: _finish_hunting 0 2024-03-29T10:53:40.545 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.563+0000 7fe57159a640 1 monclient: found mon.noname-b 2024-03-29T10:53:40.545 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.563+0000 7fe57159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:40.545 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.563+0000 7fe57159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:53:40.545 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.563+0000 7fe57159a640 1 -- 172.21.15.112:0/84652003 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5740ade90 con 0x7fe57406c1e0 2024-03-29T10:53:40.545 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.563+0000 7fe57159a640 10 monclient: _finish_auth 0 2024-03-29T10:53:40.545 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.563+0000 7fe57159a640 10 monclient: _check_auth_tickets 2024-03-29T10:53:40.545 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.563+0000 7fe57159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:40.545 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.563+0000 7fe57159a640 1 -- 172.21.15.112:0/84652003 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe564001690 con 0x7fe57406c1e0 2024-03-29T10:53:40.545 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.563+0000 7fe57159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:40.545 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.563+0000 7fe57159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:40.545 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.563+0000 7fe57159a640 10 monclient: dump: 2024-03-29T10:53:40.545 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:40.545 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:40.546 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:40.546 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:40.546 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:40.546 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:40.546 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:40.546 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:40.546 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:40.546 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:40.546 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.563+0000 7fe578a2a640 5 monclient: authenticate success, global_id 4307 2024-03-29T10:53:40.546 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.563+0000 7fe578a2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:40.546 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.563+0000 7fe57159a640 1 -- 172.21.15.112:0/84652003 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe564001860 con 0x7fe57406c1e0 2024-03-29T10:53:40.546 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.563+0000 7fe57159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:40.546 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.563+0000 7fe578a2a640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:40.547 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.563+0000 7fe578a2a640 4 set_mon_vals no callback set 2024-03-29T10:53:40.547 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.563+0000 7fe57159a640 1 -- 172.21.15.112:0/84652003 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe564001cb0 con 0x7fe57406c1e0 2024-03-29T10:53:40.547 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.563+0000 7fe57159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:40.547 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.563+0000 7fe57159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:40.547 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.563+0000 7fe57159a640 10 monclient: dump: 2024-03-29T10:53:40.547 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:40.547 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:40.547 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:40.547 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:40.547 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:40.547 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:40.547 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:40.547 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:40.547 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:40.547 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:40.554 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.565+0000 7fe578a2a640 10 monclient: shutdown 2024-03-29T10:53:40.554 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.565+0000 7fe578a2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:40.554 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.565+0000 7fe578a2a640 1 -- 172.21.15.112:0/84652003 >> v1:172.21.15.112:6790/0 conn(0x7fe57406c1e0 legacy=0x7fe574145d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:40.554 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.565+0000 7fe578a2a640 1 -- 172.21.15.112:0/84652003 shutdown_connections 2024-03-29T10:53:40.554 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.565+0000 7fe578a2a640 1 -- 172.21.15.112:0/84652003 >> 172.21.15.112:0/84652003 conn(0x7fe574067500 msgr2=0x7fe5740b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:40.554 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.565+0000 7fe578a2a640 1 -- 172.21.15.112:0/84652003 shutdown_connections 2024-03-29T10:53:40.555 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.565+0000 7fe578a2a640 1 -- 172.21.15.112:0/84652003 wait complete. 2024-03-29T10:53:40.555 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.566+0000 7fe578a2a640 10 monclient: build_initial_monmap 2024-03-29T10:53:40.555 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.566+0000 7fe578a2a640 10 monclient: monmap: 2024-03-29T10:53:40.555 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:53:40.555 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:40.555 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:53:40.555 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:53:40.555 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:40.555 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:40.555 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:53:40.555 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:53:40.555 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:53:40.555 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:40.555 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.566+0000 7fe578a2a640 1 librados: starting msgr at 2024-03-29T10:53:40.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.566+0000 7fe578a2a640 1 librados: starting objecter 2024-03-29T10:53:40.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.567+0000 7fe578a2a640 1 Processor -- start 2024-03-29T10:53:40.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.567+0000 7fe578a2a640 1 -- start start 2024-03-29T10:53:40.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.567+0000 7fe578a2a640 1 librados: setting wanted keys 2024-03-29T10:53:40.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.567+0000 7fe578a2a640 1 librados: calling monclient init 2024-03-29T10:53:40.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.567+0000 7fe578a2a640 10 monclient: init 2024-03-29T10:53:40.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.568+0000 7fe578a2a640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:40.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.568+0000 7fe578a2a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:40.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.568+0000 7fe578a2a640 10 monclient(hunting): picked mon.noname-b con 0x7fe57406c1e0 addr v1:172.21.15.118:6789/0 2024-03-29T10:53:40.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.568+0000 7fe578a2a640 10 monclient(hunting): picked mon.noname-c con 0x7fe57406c5c0 addr v1:172.21.15.112:6790/0 2024-03-29T10:53:40.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.568+0000 7fe578a2a640 10 monclient(hunting): picked mon.noname-a con 0x7fe5740ae590 addr v1:172.21.15.112:6789/0 2024-03-29T10:53:40.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.568+0000 7fe578a2a640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe574135da0 con 0x7fe5740ae590 2024-03-29T10:53:40.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.568+0000 7fe57259c640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fe57406c1e0 0x7fe5741422a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:55172/0 (socket says 172.21.15.112:55172) 2024-03-29T10:53:40.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.568+0000 7fe57259c640 1 -- 172.21.15.112:0/248851436 learned_addr learned my addr 172.21.15.112:0/248851436 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:53:40.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.568+0000 7fe578a2a640 1 -- 172.21.15.112:0/248851436 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe574141df0 con 0x7fe57406c1e0 2024-03-29T10:53:40.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.568+0000 7fe562ffd640 10 client.?.objecter ms_handle_connect 0x7fe57406c1e0 2024-03-29T10:53:40.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.568+0000 7fe562ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:40.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.568+0000 7fe578a2a640 1 -- 172.21.15.112:0/248851436 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe574135ff0 con 0x7fe57406c5c0 2024-03-29T10:53:40.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.568+0000 7fe578a2a640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:40.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.568+0000 7fe578a2a640 10 monclient(hunting): authenticate will time out at 975.872998s 2024-03-29T10:53:40.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.568+0000 7fe562ffd640 1 -- 172.21.15.112:0/248851436 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368089488 0 0) 0x7fe574141df0 con 0x7fe57406c1e0 2024-03-29T10:53:40.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.568+0000 7fe562ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:40.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.568+0000 7fe562ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:40.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.568+0000 7fe562ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:40.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.568+0000 7fe562ffd640 10 monclient(hunting): my global_id is 4312 2024-03-29T10:53:40.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.568+0000 7fe562ffd640 1 -- 172.21.15.112:0/248851436 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe554004eb0 con 0x7fe57406c1e0 2024-03-29T10:53:40.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.569+0000 7fe562ffd640 10 client.?.objecter ms_handle_connect 0x7fe57406c5c0 2024-03-29T10:53:40.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.569+0000 7fe562ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:40.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.569+0000 7fe562ffd640 1 -- 172.21.15.112:0/248851436 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357915694 0 0) 0x7fe574135ff0 con 0x7fe57406c5c0 2024-03-29T10:53:40.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.569+0000 7fe562ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:40.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.569+0000 7fe562ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:40.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.569+0000 7fe562ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:40.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.569+0000 7fe562ffd640 10 monclient(hunting): my global_id is 4310 2024-03-29T10:53:40.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.569+0000 7fe562ffd640 1 -- 172.21.15.112:0/248851436 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5540051d0 con 0x7fe57406c5c0 2024-03-29T10:53:40.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.569+0000 7fe562ffd640 1 -- 172.21.15.112:0/248851436 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3611449080 0 0) 0x7fe554004eb0 con 0x7fe57406c1e0 2024-03-29T10:53:40.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.569+0000 7fe562ffd640 1 -- 172.21.15.112:0/248851436 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe574135ff0 con 0x7fe57406c1e0 2024-03-29T10:53:40.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.569+0000 7fe562ffd640 1 -- 172.21.15.112:0/248851436 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 286667477 0 0) 0x7fe5540051d0 con 0x7fe57406c5c0 2024-03-29T10:53:40.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.569+0000 7fe562ffd640 1 -- 172.21.15.112:0/248851436 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe554005a20 con 0x7fe57406c5c0 2024-03-29T10:53:40.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.569+0000 7fe562ffd640 1 -- 172.21.15.112:0/248851436 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe568001510 con 0x7fe57406c5c0 2024-03-29T10:53:40.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.569+0000 7fe562ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:40.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.569+0000 7fe562ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:40.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.569+0000 7fe562ffd640 10 monclient(hunting): dump: 2024-03-29T10:53:40.559 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:40.559 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:40.559 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:40.559 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:40.559 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:40.559 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:40.559 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:40.559 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:40.559 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:40.559 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:40.559 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.569+0000 7fe562ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:40.559 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.570+0000 7fe562ffd640 1 -- 172.21.15.112:0/248851436 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2820113972 0 0) 0x7fe554005a20 con 0x7fe57406c5c0 2024-03-29T10:53:40.559 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.570+0000 7fe562ffd640 1 -- 172.21.15.112:0/248851436 >> v1:172.21.15.118:6789/0 conn(0x7fe57406c1e0 legacy=0x7fe5741422a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:40.559 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.570+0000 7fe562ffd640 1 -- 172.21.15.112:0/248851436 >> v1:172.21.15.112:6789/0 conn(0x7fe5740ae590 legacy=0x7fe5741e2200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:53:40.559 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.570+0000 7fe562ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:53:40.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.570+0000 7fe562ffd640 1 monclient: found mon.c 2024-03-29T10:53:40.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.570+0000 7fe562ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:40.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.570+0000 7fe562ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:53:40.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.570+0000 7fe562ffd640 1 -- 172.21.15.112:0/248851436 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5741362d0 con 0x7fe57406c5c0 2024-03-29T10:53:40.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.570+0000 7fe562ffd640 10 monclient: _finish_auth 0 2024-03-29T10:53:40.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.570+0000 7fe562ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:53:40.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.570+0000 7fe562ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:40.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.570+0000 7fe578a2a640 5 monclient: authenticate success, global_id 4310 2024-03-29T10:53:40.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.570+0000 7fe578a2a640 10 monclient: _renew_subs 2024-03-29T10:53:40.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.570+0000 7fe578a2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:53:40.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.570+0000 7fe578a2a640 1 -- 172.21.15.112:0/248851436 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe574142e20 con 0x7fe57406c5c0 2024-03-29T10:53:40.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.570+0000 7fe578a2a640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:40.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.570+0000 7fe578a2a640 10 monclient: _renew_subs 2024-03-29T10:53:40.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.570+0000 7fe578a2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:53:40.561 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.570+0000 7fe578a2a640 1 -- 172.21.15.112:0/248851436 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe574141550 con 0x7fe57406c5c0 2024-03-29T10:53:40.561 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.570+0000 7fe578a2a640 1 librados: init done 2024-03-29T10:53:40.561 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.571+0000 7fe562ffd640 1 -- 172.21.15.112:0/248851436 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe568001a20 con 0x7fe57406c5c0 2024-03-29T10:53:40.561 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.571+0000 7fe562ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:40.561 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.571+0000 7fe562ffd640 1 -- 172.21.15.112:0/248851436 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe568003910 con 0x7fe57406c5c0 2024-03-29T10:53:40.561 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.571+0000 7fe562ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:40.561 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.571+0000 7fe562ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:40.561 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.571+0000 7fe562ffd640 10 monclient: dump: 2024-03-29T10:53:40.561 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:40.561 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:40.561 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:40.561 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:40.561 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:40.561 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:40.562 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:40.562 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:40.562 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:40.562 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:40.562 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.571+0000 7fe570d99640 4 set_mon_vals no callback set 2024-03-29T10:53:40.562 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.572+0000 7fe578a2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:40.562 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.572+0000 7fe578a2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:40.562 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.572+0000 7fe578a2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:53:40.562 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.572+0000 7fe578a2a640 1 -- 172.21.15.112:0/248851436 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe558005080 con 0x7fe57406c5c0 2024-03-29T10:53:40.563 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.575+0000 7fe562ffd640 1 -- 172.21.15.112:0/248851436 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fe5680198e0 con 0x7fe57406c5c0 2024-03-29T10:53:40.563 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.576+0000 7fe562ffd640 1 -- 172.21.15.112:0/248851436 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7fe5680837b0 con 0x7fe57406c5c0 2024-03-29T10:53:40.563 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.576+0000 7fe562ffd640 10 client.4310.objecter ms_dispatch 0x7fe574000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:40.563 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.576+0000 7fe562ffd640 3 client.4310.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:40.563 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.576+0000 7fe562ffd640 3 client.4310.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:40.563 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.576+0000 7fe562ffd640 20 client.4310.objecter dump_active .. 0 homeless 2024-03-29T10:53:40.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.577+0000 7fe57259c640 10 monclient: get_auth_request con 0x7fe55406dff0 auth_method 0 2024-03-29T10:53:40.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.577+0000 7fe562ffd640 10 client.4310.objecter ms_handle_connect 0x7fe55406dff0 2024-03-29T10:53:40.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.583+0000 7fe562ffd640 1 -- 172.21.15.112:0/248851436 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe56804f0f0 con 0x7fe57406c5c0 2024-03-29T10:53:40.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.583+0000 7fe562ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:40.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.583+0000 7fe562ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:40.865 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.884+0000 7fe578a2a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:40.865 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.884+0000 7fe578a2a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:40.865 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.884+0000 7fe578a2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:53:40.865 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.884+0000 7fe578a2a640 1 -- 172.21.15.112:0/248851436 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe558005330 con 0x7fe57406c5c0 2024-03-29T10:53:40.867 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.886+0000 7fe562ffd640 1 -- 172.21.15.112:0/248851436 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7fe568082e60 con 0x7fe57406c5c0 2024-03-29T10:53:40.867 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.886+0000 7fe562ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:40.867 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.886+0000 7fe562ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:40.867 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:53:40.868 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:40.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.888+0000 7fe578a2a640 10 librados: watch_flush enter 2024-03-29T10:53:40.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.889+0000 7fe578a2a640 10 librados: watch_flush exit 2024-03-29T10:53:40.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.889+0000 7fe578a2a640 20 client.4310.objecter shutdown clearing up homeless session... 2024-03-29T10:53:40.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.889+0000 7fe578a2a640 10 client.4310.objecter successfully canceled tick 2024-03-29T10:53:40.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.889+0000 7fe578a2a640 1 -- 172.21.15.112:0/248851436 >> v1:172.21.15.118:6816/3181073408 conn(0x7fe55406dff0 legacy=0x7fe5540704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:40.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.889+0000 7fe578a2a640 10 monclient: shutdown 2024-03-29T10:53:40.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.889+0000 7fe578a2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:40.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.889+0000 7fe578a2a640 1 -- 172.21.15.112:0/248851436 >> v1:172.21.15.112:6790/0 conn(0x7fe57406c5c0 legacy=0x7fe5741366f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:40.871 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.889+0000 7fe578a2a640 1 -- 172.21.15.112:0/248851436 shutdown_connections 2024-03-29T10:53:40.871 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.889+0000 7fe578a2a640 1 -- 172.21.15.112:0/248851436 >> 172.21.15.112:0/248851436 conn(0x7fe574067500 msgr2=0x7fe5740b24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:40.871 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.889+0000 7fe578a2a640 1 -- 172.21.15.112:0/248851436 shutdown_connections 2024-03-29T10:53:40.871 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.889+0000 7fe578a2a640 1 -- 172.21.15.112:0/248851436 wait complete. 2024-03-29T10:53:40.871 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:40.889+0000 7fe578a2a640 1 librados: shutdown 2024-03-29T10:53:40.884 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:47.691 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:47.836 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.855+0000 7fce18d6e640 10 monclient: get_monmap_and_config 2024-03-29T10:53:47.836 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.855+0000 7fce18d6e640 10 monclient: build_initial_monmap 2024-03-29T10:53:47.836 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.855+0000 7fce18d6e640 10 monclient: monmap: 2024-03-29T10:53:47.836 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:53:47.836 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:47.837 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:53:47.856183+0000 2024-03-29T10:53:47.837 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:53:47.856183+0000 2024-03-29T10:53:47.837 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:47.837 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:47.837 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:53:47.837 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:53:47.837 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:53:47.837 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:47.838 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.856+0000 7fce18d6e640 1 Processor -- start 2024-03-29T10:53:47.838 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.856+0000 7fce18d6e640 1 -- start start 2024-03-29T10:53:47.838 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.856+0000 7fce18d6e640 10 monclient: init 2024-03-29T10:53:47.838 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.856+0000 7fce18d6e640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:47.838 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.856+0000 7fce18d6e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:47.838 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.856+0000 7fce18d6e640 10 monclient(hunting): picked mon.noname-a con 0x7fce0c0389e0 addr v1:172.21.15.112:6789/0 2024-03-29T10:53:47.838 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.856+0000 7fce18d6e640 10 monclient(hunting): picked mon.noname-b con 0x7fce0c038dc0 addr v1:172.21.15.112:6790/0 2024-03-29T10:53:47.838 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.856+0000 7fce18d6e640 10 monclient(hunting): picked mon.noname-c con 0x7fce0c030660 addr v1:172.21.15.118:6789/0 2024-03-29T10:53:47.838 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.856+0000 7fce18d6e640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0c0c54c0 con 0x7fce0c0389e0 2024-03-29T10:53:47.838 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.856+0000 7fce18d6e640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0c036430 con 0x7fce0c030660 2024-03-29T10:53:47.838 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.857+0000 7fce18d6e640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0c039840 con 0x7fce0c038dc0 2024-03-29T10:53:47.838 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.857+0000 7fce18d6e640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:47.839 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.857+0000 7fce18d6e640 10 monclient(hunting): authenticate will time out at 983.161190s 2024-03-29T10:53:47.841 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.860+0000 7fce12ffd640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fce0c038dc0 0x7fce0c0cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59339/0 (socket says 172.21.15.112:59339) 2024-03-29T10:53:47.841 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.860+0000 7fce12ffd640 1 -- 172.21.15.112:0/3997257130 learned_addr learned my addr 172.21.15.112:0/3997257130 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:53:47.842 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.860+0000 7fce127fc640 1 -- 172.21.15.112:0/3997257130 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152095174 0 0) 0x7fce0c039840 con 0x7fce0c038dc0 2024-03-29T10:53:47.842 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.860+0000 7fce127fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:47.842 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.860+0000 7fce127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:47.842 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.860+0000 7fce127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:47.842 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.860+0000 7fce127fc640 10 monclient(hunting): my global_id is 4313 2024-03-29T10:53:47.842 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.860+0000 7fce127fc640 1 -- 172.21.15.112:0/3997257130 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce04003520 con 0x7fce0c038dc0 2024-03-29T10:53:47.843 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.860+0000 7fce127fc640 1 -- 172.21.15.112:0/3997257130 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2763616355 0 0) 0x7fce0c0c54c0 con 0x7fce0c0389e0 2024-03-29T10:53:47.843 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.860+0000 7fce127fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:47.843 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.860+0000 7fce127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:47.843 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.860+0000 7fce127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:47.843 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.860+0000 7fce127fc640 10 monclient(hunting): my global_id is 4311 2024-03-29T10:53:47.843 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.860+0000 7fce127fc640 1 -- 172.21.15.112:0/3997257130 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce0c039840 con 0x7fce0c0389e0 2024-03-29T10:53:47.843 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.860+0000 7fce127fc640 1 -- 172.21.15.112:0/3997257130 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1699137359 0 0) 0x7fce04003520 con 0x7fce0c038dc0 2024-03-29T10:53:47.843 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.861+0000 7fce127fc640 1 -- 172.21.15.112:0/3997257130 >> v1:172.21.15.118:6789/0 conn(0x7fce0c030660 legacy=0x7fce0c0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:53:47.843 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.861+0000 7fce127fc640 1 -- 172.21.15.112:0/3997257130 >> v1:172.21.15.112:6789/0 conn(0x7fce0c0389e0 legacy=0x7fce0c033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:47.843 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.861+0000 7fce127fc640 10 monclient: _finish_hunting 0 2024-03-29T10:53:47.843 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.861+0000 7fce127fc640 1 monclient: found mon.noname-b 2024-03-29T10:53:47.843 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.861+0000 7fce127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:47.843 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.861+0000 7fce127fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:53:47.844 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.861+0000 7fce127fc640 1 -- 172.21.15.112:0/3997257130 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0c035690 con 0x7fce0c038dc0 2024-03-29T10:53:47.844 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.861+0000 7fce127fc640 10 monclient: _finish_auth 0 2024-03-29T10:53:47.844 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.861+0000 7fce127fc640 10 monclient: _check_auth_tickets 2024-03-29T10:53:47.844 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.861+0000 7fce127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:47.844 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.861+0000 7fce18d6e640 5 monclient: authenticate success, global_id 4313 2024-03-29T10:53:47.844 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.861+0000 7fce18d6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:47.844 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.861+0000 7fce127fc640 1 -- 172.21.15.112:0/3997257130 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fce00001690 con 0x7fce0c038dc0 2024-03-29T10:53:47.844 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.861+0000 7fce127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:47.844 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.861+0000 7fce127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:47.844 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.861+0000 7fce127fc640 10 monclient: dump: 2024-03-29T10:53:47.844 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:47.844 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:47.844 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:47.844 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:47.845 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:47.845 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:47.845 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:47.845 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:47.845 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:47.845 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:47.845 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.861+0000 7fce18d6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:47.845 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.861+0000 7fce127fc640 1 -- 172.21.15.112:0/3997257130 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce000017f0 con 0x7fce0c038dc0 2024-03-29T10:53:47.845 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.861+0000 7fce127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:47.845 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.861+0000 7fce18d6e640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:47.845 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.861+0000 7fce18d6e640 4 set_mon_vals no callback set 2024-03-29T10:53:47.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.862+0000 7fce127fc640 1 -- 172.21.15.112:0/3997257130 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fce00001c00 con 0x7fce0c038dc0 2024-03-29T10:53:47.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.862+0000 7fce127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:47.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.862+0000 7fce127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:47.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.862+0000 7fce127fc640 10 monclient: dump: 2024-03-29T10:53:47.852 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:47.852 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:47.852 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:47.853 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:47.853 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:47.853 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:47.853 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:47.853 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:47.853 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:47.853 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:47.853 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.862+0000 7fce18d6e640 10 monclient: shutdown 2024-03-29T10:53:47.853 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.862+0000 7fce18d6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:47.853 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.862+0000 7fce18d6e640 1 -- 172.21.15.112:0/3997257130 >> v1:172.21.15.112:6790/0 conn(0x7fce0c038dc0 legacy=0x7fce0c0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:47.853 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.862+0000 7fce18d6e640 1 -- 172.21.15.112:0/3997257130 shutdown_connections 2024-03-29T10:53:47.853 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.863+0000 7fce18d6e640 1 -- 172.21.15.112:0/3997257130 >> 172.21.15.112:0/3997257130 conn(0x7fce0c005c00 msgr2=0x7fce0c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:47.854 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.863+0000 7fce18d6e640 1 -- 172.21.15.112:0/3997257130 shutdown_connections 2024-03-29T10:53:47.854 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.863+0000 7fce18d6e640 1 -- 172.21.15.112:0/3997257130 wait complete. 2024-03-29T10:53:47.854 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.864+0000 7fce18d6e640 10 monclient: build_initial_monmap 2024-03-29T10:53:47.854 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.864+0000 7fce18d6e640 10 monclient: monmap: 2024-03-29T10:53:47.854 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:53:47.854 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:47.854 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:53:47.854 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:53:47.854 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:47.854 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:47.854 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:53:47.854 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:53:47.854 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:53:47.854 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:47.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.864+0000 7fce18d6e640 1 librados: starting msgr at 2024-03-29T10:53:47.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.864+0000 7fce18d6e640 1 librados: starting objecter 2024-03-29T10:53:47.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.864+0000 7fce18d6e640 1 Processor -- start 2024-03-29T10:53:47.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.865+0000 7fce18d6e640 1 -- start start 2024-03-29T10:53:47.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.865+0000 7fce18d6e640 1 librados: setting wanted keys 2024-03-29T10:53:47.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.865+0000 7fce18d6e640 1 librados: calling monclient init 2024-03-29T10:53:47.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.865+0000 7fce18d6e640 10 monclient: init 2024-03-29T10:53:47.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.865+0000 7fce18d6e640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:47.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.865+0000 7fce18d6e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:53:47.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.865+0000 7fce18d6e640 10 monclient(hunting): picked mon.noname-b con 0x7fce0c030660 addr v1:172.21.15.118:6789/0 2024-03-29T10:53:47.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.865+0000 7fce18d6e640 10 monclient(hunting): picked mon.noname-a con 0x7fce0c0389e0 addr v1:172.21.15.112:6789/0 2024-03-29T10:53:47.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.867+0000 7fce137fe640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fce0c030660 0x7fce0c0c24e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:56040/0 (socket says 172.21.15.112:56040) 2024-03-29T10:53:47.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.868+0000 7fce18d6e640 10 monclient(hunting): picked mon.noname-c con 0x7fce0c0c7d70 addr v1:172.21.15.112:6790/0 2024-03-29T10:53:47.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.868+0000 7fce18d6e640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0c0a59f0 con 0x7fce0c0389e0 2024-03-29T10:53:47.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.868+0000 7fce18d6e640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0c0c82e0 con 0x7fce0c030660 2024-03-29T10:53:47.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.868+0000 7fce18d6e640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0c0c2bf0 con 0x7fce0c0c7d70 2024-03-29T10:53:47.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.868+0000 7fce18d6e640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:47.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.868+0000 7fce18d6e640 10 monclient(hunting): authenticate will time out at 983.172798s 2024-03-29T10:53:47.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.869+0000 7fce137fe640 1 -- 172.21.15.112:0/1970219276 learned_addr learned my addr 172.21.15.112:0/1970219276 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:53:47.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.869+0000 7fcdfffff640 10 client.?.objecter ms_handle_connect 0x7fce0c030660 2024-03-29T10:53:47.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.869+0000 7fcdfffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:47.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.869+0000 7fcdfffff640 1 -- 172.21.15.112:0/1970219276 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191243435 0 0) 0x7fce0c0c82e0 con 0x7fce0c030660 2024-03-29T10:53:47.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.869+0000 7fcdfffff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:47.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.869+0000 7fcdfffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:47.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.869+0000 7fcdfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:47.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.869+0000 7fcdfffff640 10 monclient(hunting): my global_id is 4315 2024-03-29T10:53:47.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.869+0000 7fcdfffff640 1 -- 172.21.15.112:0/1970219276 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdf8004eb0 con 0x7fce0c030660 2024-03-29T10:53:47.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.870+0000 7fcdfffff640 1 -- 172.21.15.112:0/1970219276 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1853151270 0 0) 0x7fcdf8004eb0 con 0x7fce0c030660 2024-03-29T10:53:47.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.870+0000 7fcdfffff640 1 -- 172.21.15.112:0/1970219276 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce0c0c82e0 con 0x7fce0c030660 2024-03-29T10:53:47.857 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.870+0000 7fcdfffff640 1 -- 172.21.15.112:0/1970219276 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fce08001a10 con 0x7fce0c030660 2024-03-29T10:53:47.857 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.870+0000 7fcdfffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:47.857 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.870+0000 7fcdfffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:47.857 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.870+0000 7fcdfffff640 10 monclient(hunting): dump: 2024-03-29T10:53:47.857 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:47.857 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:47.857 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:47.857 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:47.857 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:47.857 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:47.857 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:47.857 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:47.857 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:47.857 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:47.857 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.870+0000 7fcdfffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:47.858 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.870+0000 7fcdfffff640 1 -- 172.21.15.112:0/1970219276 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 570637383 0 0) 0x7fce0c0c82e0 con 0x7fce0c030660 2024-03-29T10:53:47.858 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.870+0000 7fcdfffff640 1 -- 172.21.15.112:0/1970219276 >> v1:172.21.15.112:6790/0 conn(0x7fce0c0c7d70 legacy=0x7fce0c168280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:47.858 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.870+0000 7fcdfffff640 1 -- 172.21.15.112:0/1970219276 >> v1:172.21.15.112:6789/0 conn(0x7fce0c0389e0 legacy=0x7fce0c15ca00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:53:47.858 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.871+0000 7fcdfffff640 10 monclient: _finish_hunting 0 2024-03-29T10:53:47.858 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.871+0000 7fcdfffff640 1 monclient: found mon.b 2024-03-29T10:53:47.858 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.871+0000 7fcdfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:47.858 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.871+0000 7fcdfffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:53:47.858 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.871+0000 7fcdfffff640 1 -- 172.21.15.112:0/1970219276 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0c0c2d80 con 0x7fce0c030660 2024-03-29T10:53:47.858 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.871+0000 7fcdfffff640 10 monclient: _finish_auth 0 2024-03-29T10:53:47.858 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.871+0000 7fcdfffff640 10 monclient: _check_auth_tickets 2024-03-29T10:53:47.858 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.871+0000 7fcdfffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:47.858 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.871+0000 7fce18d6e640 5 monclient: authenticate success, global_id 4315 2024-03-29T10:53:47.858 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.871+0000 7fce18d6e640 10 monclient: _renew_subs 2024-03-29T10:53:47.858 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.871+0000 7fce18d6e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:53:47.859 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.871+0000 7fce18d6e640 1 -- 172.21.15.112:0/1970219276 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce0c02f500 con 0x7fce0c030660 2024-03-29T10:53:47.859 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.871+0000 7fce18d6e640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:47.859 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.871+0000 7fce18d6e640 10 monclient: _renew_subs 2024-03-29T10:53:47.859 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.871+0000 7fce18d6e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:53:47.859 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.871+0000 7fce18d6e640 1 -- 172.21.15.112:0/1970219276 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce0c0c75d0 con 0x7fce0c030660 2024-03-29T10:53:47.859 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.871+0000 7fce18d6e640 1 librados: init done 2024-03-29T10:53:47.860 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.873+0000 7fcdfffff640 1 -- 172.21.15.112:0/1970219276 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce08001b70 con 0x7fce0c030660 2024-03-29T10:53:47.861 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.874+0000 7fce18d6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:47.861 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.874+0000 7fce18d6e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:47.861 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.874+0000 7fce18d6e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:53:47.861 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.874+0000 7fce18d6e640 1 -- 172.21.15.112:0/1970219276 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce0c007b90 con 0x7fce0c030660 2024-03-29T10:53:47.861 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.874+0000 7fcdfffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:47.861 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.874+0000 7fcdfffff640 1 -- 172.21.15.112:0/1970219276 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fce08003a60 con 0x7fce0c030660 2024-03-29T10:53:47.861 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.874+0000 7fcdfffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:47.861 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.874+0000 7fcdfffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:47.861 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.874+0000 7fcdfffff640 10 monclient: dump: 2024-03-29T10:53:47.861 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:47.861 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:47.861 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:47.861 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:47.861 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:47.862 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:47.862 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:47.862 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:47.862 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:47.862 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:47.862 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.874+0000 7fce11ffb640 4 set_mon_vals no callback set 2024-03-29T10:53:47.862 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.874+0000 7fcdfffff640 1 -- 172.21.15.112:0/1970219276 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fce08003c80 con 0x7fce0c030660 2024-03-29T10:53:47.862 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.875+0000 7fcdfffff640 1 -- 172.21.15.112:0/1970219276 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7fce08084b50 con 0x7fce0c030660 2024-03-29T10:53:47.862 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.875+0000 7fcdfffff640 10 client.4315.objecter ms_dispatch 0x7fce14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:47.862 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.875+0000 7fcdfffff640 3 client.4315.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:47.862 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.875+0000 7fcdfffff640 3 client.4315.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:47.862 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.876+0000 7fcdfffff640 20 client.4315.objecter dump_active .. 0 homeless 2024-03-29T10:53:47.862 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.876+0000 7fce12ffd640 10 monclient: get_auth_request con 0x7fcdf806dc10 auth_method 0 2024-03-29T10:53:47.862 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.876+0000 7fcdfffff640 10 client.4315.objecter ms_handle_connect 0x7fcdf806dc10 2024-03-29T10:53:47.864 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.883+0000 7fcdfffff640 1 -- 172.21.15.112:0/1970219276 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fce08050550 con 0x7fce0c030660 2024-03-29T10:53:47.864 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.883+0000 7fcdfffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:47.864 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:47.883+0000 7fcdfffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:48.184 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:48.203+0000 7fce18d6e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:48.185 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:48.203+0000 7fce18d6e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:48.185 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:48.203+0000 7fce18d6e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:53:48.185 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:48.203+0000 7fce18d6e640 1 -- 172.21.15.112:0/1970219276 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce0c0c8520 con 0x7fce0c030660 2024-03-29T10:53:48.185 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:48.204+0000 7fcdfffff640 1 -- 172.21.15.112:0/1970219276 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7fce0c0c8520 con 0x7fce0c030660 2024-03-29T10:53:48.185 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:48.204+0000 7fcdfffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:48.185 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:48.204+0000 7fcdfffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:48.186 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:53:48.186 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:48.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:48.207+0000 7fce18d6e640 10 librados: watch_flush enter 2024-03-29T10:53:48.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:48.208+0000 7fce18d6e640 10 librados: watch_flush exit 2024-03-29T10:53:48.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:48.208+0000 7fce18d6e640 20 client.4315.objecter shutdown clearing up homeless session... 2024-03-29T10:53:48.192 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:48.208+0000 7fce18d6e640 10 client.4315.objecter successfully canceled tick 2024-03-29T10:53:48.192 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:48.208+0000 7fce18d6e640 1 -- 172.21.15.112:0/1970219276 >> v1:172.21.15.118:6816/3181073408 conn(0x7fcdf806dc10 legacy=0x7fcdf80700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:48.192 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:48.208+0000 7fce18d6e640 10 monclient: shutdown 2024-03-29T10:53:48.192 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:48.208+0000 7fce18d6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:48.192 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:48.209+0000 7fce18d6e640 1 -- 172.21.15.112:0/1970219276 >> v1:172.21.15.118:6789/0 conn(0x7fce0c030660 legacy=0x7fce0c0c24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:48.192 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:48.209+0000 7fce18d6e640 1 -- 172.21.15.112:0/1970219276 shutdown_connections 2024-03-29T10:53:48.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:48.209+0000 7fce18d6e640 1 -- 172.21.15.112:0/1970219276 >> 172.21.15.112:0/1970219276 conn(0x7fce0c005c00 msgr2=0x7fce0c034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:48.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:48.209+0000 7fce18d6e640 1 -- 172.21.15.112:0/1970219276 shutdown_connections 2024-03-29T10:53:48.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:48.209+0000 7fce18d6e640 1 -- 172.21.15.112:0/1970219276 wait complete. 2024-03-29T10:53:48.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:48.209+0000 7fce18d6e640 1 librados: shutdown 2024-03-29T10:53:48.207 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:55.014 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:55.166 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.183+0000 7f0bccbcd640 10 monclient: get_monmap_and_config 2024-03-29T10:53:55.166 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.183+0000 7f0bccbcd640 10 monclient: build_initial_monmap 2024-03-29T10:53:55.167 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.183+0000 7f0bccbcd640 10 monclient: monmap: 2024-03-29T10:53:55.167 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:53:55.167 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:55.167 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:53:55.185755+0000 2024-03-29T10:53:55.167 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:53:55.185755+0000 2024-03-29T10:53:55.167 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:55.167 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:55.167 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:53:55.167 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:53:55.167 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:53:55.167 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:55.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.184+0000 7f0bccbcd640 1 Processor -- start 2024-03-29T10:53:55.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.184+0000 7f0bccbcd640 1 -- start start 2024-03-29T10:53:55.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.184+0000 7f0bccbcd640 10 monclient: init 2024-03-29T10:53:55.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.185+0000 7f0bccbcd640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:55.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.185+0000 7f0bccbcd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:53:55.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.186+0000 7f0bccbcd640 10 monclient(hunting): picked mon.noname-b con 0x7f0bc0034850 addr v1:172.21.15.112:6790/0 2024-03-29T10:53:55.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.186+0000 7f0bccbcd640 10 monclient(hunting): picked mon.noname-a con 0x7f0bc002a640 addr v1:172.21.15.112:6789/0 2024-03-29T10:53:55.171 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.186+0000 7f0bccbcd640 10 monclient(hunting): picked mon.noname-c con 0x7f0bc00327d0 addr v1:172.21.15.118:6789/0 2024-03-29T10:53:55.171 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.189+0000 7f0bccbcd640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bc002c820 con 0x7f0bc002a640 2024-03-29T10:53:55.171 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.189+0000 7f0bccbcd640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bc002a120 con 0x7f0bc00327d0 2024-03-29T10:53:55.171 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.189+0000 7f0bccbcd640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bc00c75b0 con 0x7f0bc0034850 2024-03-29T10:53:55.172 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.189+0000 7f0bccbcd640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:55.172 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.189+0000 7f0bccbcd640 10 monclient(hunting): authenticate will time out at 990.494696s 2024-03-29T10:53:55.172 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.189+0000 7f0bc6d9d640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f0bc00327d0 0x7f0bc00cf240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:57061/0 (socket says 172.21.15.112:57061) 2024-03-29T10:53:55.172 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.189+0000 7f0bc6d9d640 1 -- 172.21.15.112:0/3112711207 learned_addr learned my addr 172.21.15.112:0/3112711207 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:53:55.174 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.192+0000 7f0bc559a640 1 -- 172.21.15.112:0/3112711207 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3409192081 0 0) 0x7f0bc002a120 con 0x7f0bc00327d0 2024-03-29T10:53:55.174 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.192+0000 7f0bc559a640 10 monclient(hunting): _negotiate 2024-03-29T10:53:55.174 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.192+0000 7f0bc559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:55.174 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.192+0000 7f0bc559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:55.174 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.192+0000 7f0bc559a640 10 monclient(hunting): my global_id is 4318 2024-03-29T10:53:55.174 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.192+0000 7f0bc559a640 1 -- 172.21.15.112:0/3112711207 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bb0003520 con 0x7f0bc00327d0 2024-03-29T10:53:55.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.192+0000 7f0bc559a640 1 -- 172.21.15.112:0/3112711207 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1607240203 0 0) 0x7f0bc00c75b0 con 0x7f0bc0034850 2024-03-29T10:53:55.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.192+0000 7f0bc559a640 10 monclient(hunting): _negotiate 2024-03-29T10:53:55.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.192+0000 7f0bc559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:55.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.192+0000 7f0bc559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:55.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.192+0000 7f0bc559a640 10 monclient(hunting): my global_id is 4316 2024-03-29T10:53:55.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.192+0000 7f0bc559a640 1 -- 172.21.15.112:0/3112711207 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bc002a120 con 0x7f0bc0034850 2024-03-29T10:53:55.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 1 -- 172.21.15.112:0/3112711207 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653110391 0 0) 0x7f0bc002c820 con 0x7f0bc002a640 2024-03-29T10:53:55.176 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 10 monclient(hunting): _negotiate 2024-03-29T10:53:55.176 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:55.176 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:55.176 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 10 monclient(hunting): my global_id is 4314 2024-03-29T10:53:55.176 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 1 -- 172.21.15.112:0/3112711207 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bc00c75b0 con 0x7f0bc002a640 2024-03-29T10:53:55.176 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 1 -- 172.21.15.112:0/3112711207 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2682645159 0 0) 0x7f0bc002a120 con 0x7f0bc0034850 2024-03-29T10:53:55.176 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 1 -- 172.21.15.112:0/3112711207 >> v1:172.21.15.118:6789/0 conn(0x7f0bc00327d0 legacy=0x7f0bc00cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:55.176 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 1 -- 172.21.15.112:0/3112711207 >> v1:172.21.15.112:6789/0 conn(0x7f0bc002a640 legacy=0x7f0bc00320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:55.176 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 10 monclient: _finish_hunting 0 2024-03-29T10:53:55.176 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 1 monclient: found mon.noname-b 2024-03-29T10:53:55.176 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:55.176 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:53:55.176 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 1 -- 172.21.15.112:0/3112711207 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc0033110 con 0x7f0bc0034850 2024-03-29T10:53:55.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 10 monclient: _finish_auth 0 2024-03-29T10:53:55.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 10 monclient: _check_auth_tickets 2024-03-29T10:53:55.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:55.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 1 -- 172.21.15.112:0/3112711207 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f0bc804f730 con 0x7f0bc0034850 2024-03-29T10:53:55.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:55.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:55.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 10 monclient: dump: 2024-03-29T10:53:55.177 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:55.177 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:55.177 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:55.177 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:55.177 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:55.177 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:55.177 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:55.177 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:55.178 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:55.178 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:55.178 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 1 -- 172.21.15.112:0/3112711207 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0bc8075f10 con 0x7f0bc0034850 2024-03-29T10:53:55.178 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:55.178 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 1 -- 172.21.15.112:0/3112711207 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f0bc80758b0 con 0x7f0bc0034850 2024-03-29T10:53:55.178 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:55.178 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:55.178 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bc559a640 10 monclient: dump: 2024-03-29T10:53:55.178 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:55.178 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:55.178 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:55.178 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:55.178 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:55.178 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:55.179 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:55.179 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:55.179 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:55.179 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:55.179 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bccbcd640 5 monclient: authenticate success, global_id 4316 2024-03-29T10:53:55.179 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bccbcd640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:55.179 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.193+0000 7f0bccbcd640 4 set_mon_vals no callback set 2024-03-29T10:53:55.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.194+0000 7f0bccbcd640 10 monclient: shutdown 2024-03-29T10:53:55.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.195+0000 7f0bccbcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:55.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.195+0000 7f0bccbcd640 1 -- 172.21.15.112:0/3112711207 >> v1:172.21.15.112:6790/0 conn(0x7f0bc0034850 legacy=0x7f0bc00c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:55.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.197+0000 7f0bccbcd640 1 -- 172.21.15.112:0/3112711207 shutdown_connections 2024-03-29T10:53:55.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.197+0000 7f0bccbcd640 1 -- 172.21.15.112:0/3112711207 >> 172.21.15.112:0/3112711207 conn(0x7f0bc00058f0 msgr2=0x7f0bc0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:55.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.197+0000 7f0bccbcd640 1 -- 172.21.15.112:0/3112711207 shutdown_connections 2024-03-29T10:53:55.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.198+0000 7f0bccbcd640 1 -- 172.21.15.112:0/3112711207 wait complete. 2024-03-29T10:53:55.181 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.199+0000 7f0bccbcd640 10 monclient: build_initial_monmap 2024-03-29T10:53:55.181 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.199+0000 7f0bccbcd640 10 monclient: monmap: 2024-03-29T10:53:55.181 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:53:55.181 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:55.181 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:53:55.181 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:53:55.181 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:55.181 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:55.181 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:53:55.181 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:53:55.181 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:53:55.182 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:55.182 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.199+0000 7f0bccbcd640 1 librados: starting msgr at 2024-03-29T10:53:55.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.200+0000 7f0bccbcd640 1 librados: starting objecter 2024-03-29T10:53:55.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.200+0000 7f0bccbcd640 1 Processor -- start 2024-03-29T10:53:55.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.200+0000 7f0bccbcd640 1 -- start start 2024-03-29T10:53:55.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.200+0000 7f0bccbcd640 1 librados: setting wanted keys 2024-03-29T10:53:55.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.200+0000 7f0bccbcd640 1 librados: calling monclient init 2024-03-29T10:53:55.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.200+0000 7f0bccbcd640 10 monclient: init 2024-03-29T10:53:55.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.201+0000 7f0bccbcd640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:55.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.201+0000 7f0bccbcd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:53:55.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.201+0000 7f0bccbcd640 10 monclient(hunting): picked mon.noname-a con 0x7f0bc002a640 addr v1:172.21.15.112:6789/0 2024-03-29T10:53:55.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.205+0000 7f0bccbcd640 10 monclient(hunting): picked mon.noname-b con 0x7f0bc00327d0 addr v1:172.21.15.118:6789/0 2024-03-29T10:53:55.198 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.206+0000 7f0bccbcd640 10 monclient(hunting): picked mon.noname-c con 0x7f0bc00bfef0 addr v1:172.21.15.112:6790/0 2024-03-29T10:53:55.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.206+0000 7f0bccbcd640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bc00c40e0 con 0x7f0bc002a640 2024-03-29T10:53:55.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.206+0000 7f0bccbcd640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bc00c4270 con 0x7f0bc00327d0 2024-03-29T10:53:55.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.206+0000 7f0bccbcd640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bc00c4400 con 0x7f0bc00bfef0 2024-03-29T10:53:55.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.206+0000 7f0bccbcd640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:55.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.206+0000 7f0bccbcd640 10 monclient(hunting): authenticate will time out at 990.511574s 2024-03-29T10:53:55.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.209+0000 7f0bc6d9d640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f0bc00bfef0 0x7f0bc0158450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:52863/0 (socket says 172.21.15.112:52863) 2024-03-29T10:53:55.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.209+0000 7f0bc6d9d640 1 -- 172.21.15.112:0/281848509 learned_addr learned my addr 172.21.15.112:0/281848509 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:53:55.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.206+0000 7f0bc659c640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f0bc002a640 0x7f0bc00310b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:49447/0 (socket says 172.21.15.112:49447) 2024-03-29T10:53:55.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.209+0000 7f0bb6ffd640 10 client.?.objecter ms_handle_connect 0x7f0bc00bfef0 2024-03-29T10:53:55.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.209+0000 7f0bb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:55.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.209+0000 7f0bb6ffd640 10 client.?.objecter ms_handle_connect 0x7f0bc002a640 2024-03-29T10:53:55.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.209+0000 7f0bb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:55.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.210+0000 7f0bb6ffd640 1 -- 172.21.15.112:0/281848509 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59314938 0 0) 0x7f0bc00c4400 con 0x7f0bc00bfef0 2024-03-29T10:53:55.200 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.210+0000 7f0bb6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:55.200 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.210+0000 7f0bb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:55.200 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.210+0000 7f0bb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:55.200 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.210+0000 7f0bb6ffd640 10 monclient(hunting): my global_id is 4319 2024-03-29T10:53:55.200 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.210+0000 7f0bb6ffd640 1 -- 172.21.15.112:0/281848509 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ba8004eb0 con 0x7f0bc00bfef0 2024-03-29T10:53:55.200 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.210+0000 7f0bb6ffd640 1 -- 172.21.15.112:0/281848509 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1856805798 0 0) 0x7f0bc00c40e0 con 0x7f0bc002a640 2024-03-29T10:53:55.200 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.210+0000 7f0bb6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:55.200 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.210+0000 7f0bb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:55.200 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.210+0000 7f0bb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:55.200 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.210+0000 7f0bb6ffd640 10 monclient(hunting): my global_id is 4317 2024-03-29T10:53:55.200 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.210+0000 7f0bb6ffd640 1 -- 172.21.15.112:0/281848509 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bc00c4400 con 0x7f0bc002a640 2024-03-29T10:53:55.200 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.211+0000 7f0bb6ffd640 1 -- 172.21.15.112:0/281848509 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3080291632 0 0) 0x7f0ba8004eb0 con 0x7f0bc00bfef0 2024-03-29T10:53:55.200 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.211+0000 7f0bb6ffd640 1 -- 172.21.15.112:0/281848509 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0bc00c40e0 con 0x7f0bc00bfef0 2024-03-29T10:53:55.200 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.211+0000 7f0bb6ffd640 1 -- 172.21.15.112:0/281848509 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f0bbc001750 con 0x7f0bc00bfef0 2024-03-29T10:53:55.200 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.211+0000 7f0bb6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:55.201 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.211+0000 7f0bb6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:55.201 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.211+0000 7f0bb6ffd640 10 monclient(hunting): dump: 2024-03-29T10:53:55.201 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:55.201 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:55.201 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:55.201 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:55.201 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:55.201 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:55.201 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:55.201 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:55.201 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:55.201 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:55.201 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.211+0000 7f0bb6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:55.201 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.211+0000 7f0bb6ffd640 1 -- 172.21.15.112:0/281848509 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 711798159 0 0) 0x7f0bc00c40e0 con 0x7f0bc00bfef0 2024-03-29T10:53:55.201 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.211+0000 7f0bb6ffd640 1 -- 172.21.15.112:0/281848509 >> v1:172.21.15.118:6789/0 conn(0x7f0bc00327d0 legacy=0x7f0bc00bf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:55.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.211+0000 7f0bb6ffd640 1 -- 172.21.15.112:0/281848509 >> v1:172.21.15.112:6789/0 conn(0x7f0bc002a640 legacy=0x7f0bc00310b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:55.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.211+0000 7f0bb6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:53:55.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.211+0000 7f0bb6ffd640 1 monclient: found mon.c 2024-03-29T10:53:55.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.212+0000 7f0bb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:55.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.212+0000 7f0bb6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:53:55.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.212+0000 7f0bb6ffd640 1 -- 172.21.15.112:0/281848509 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc00bf900 con 0x7f0bc00bfef0 2024-03-29T10:53:55.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.212+0000 7f0bb6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:53:55.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.212+0000 7f0bb6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:53:55.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.212+0000 7f0bb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:55.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.212+0000 7f0bccbcd640 5 monclient: authenticate success, global_id 4319 2024-03-29T10:53:55.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.212+0000 7f0bccbcd640 10 monclient: _renew_subs 2024-03-29T10:53:55.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.212+0000 7f0bccbcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:53:55.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.212+0000 7f0bccbcd640 1 -- 172.21.15.112:0/281848509 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc00bfb50 con 0x7f0bc00bfef0 2024-03-29T10:53:55.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.212+0000 7f0bccbcd640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:55.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.212+0000 7f0bccbcd640 10 monclient: _renew_subs 2024-03-29T10:53:55.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.212+0000 7f0bccbcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:53:55.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.212+0000 7f0bccbcd640 1 -- 172.21.15.112:0/281848509 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc0158d40 con 0x7f0bc00bfef0 2024-03-29T10:53:55.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.212+0000 7f0bccbcd640 1 librados: init done 2024-03-29T10:53:55.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.213+0000 7f0bb6ffd640 1 -- 172.21.15.112:0/281848509 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0bbc0018b0 con 0x7f0bc00bfef0 2024-03-29T10:53:55.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.213+0000 7f0bb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:55.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.213+0000 7f0bb6ffd640 1 -- 172.21.15.112:0/281848509 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f0bbc004980 con 0x7f0bc00bfef0 2024-03-29T10:53:55.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.213+0000 7f0bb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:55.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.213+0000 7f0bb6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:55.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.213+0000 7f0bb6ffd640 10 monclient: dump: 2024-03-29T10:53:55.203 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:53:55.203 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:53:55.203 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:55.203 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:53:55.203 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:55.203 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:53:55.204 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:53:55.204 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:53:55.204 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:53:55.204 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:53:55.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.216+0000 7f0bc4d99640 4 set_mon_vals no callback set 2024-03-29T10:53:55.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.216+0000 7f0bccbcd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:55.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.217+0000 7f0bccbcd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:55.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.217+0000 7f0bccbcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:53:55.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.217+0000 7f0bb6ffd640 1 -- 172.21.15.112:0/281848509 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f0bbc004ae0 con 0x7f0bc00bfef0 2024-03-29T10:53:55.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.217+0000 7f0bccbcd640 1 -- 172.21.15.112:0/281848509 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bac005080 con 0x7f0bc00bfef0 2024-03-29T10:53:55.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.218+0000 7f0bb6ffd640 1 -- 172.21.15.112:0/281848509 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f0bbc084a20 con 0x7f0bc00bfef0 2024-03-29T10:53:55.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.218+0000 7f0bb6ffd640 10 client.4319.objecter ms_dispatch 0x7f0bc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:55.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.218+0000 7f0bb6ffd640 3 client.4319.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:55.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.218+0000 7f0bb6ffd640 3 client.4319.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:55.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.218+0000 7f0bb6ffd640 20 client.4319.objecter dump_active .. 0 homeless 2024-03-29T10:53:55.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.220+0000 7f0bc659c640 10 monclient: get_auth_request con 0x7f0ba806dd10 auth_method 0 2024-03-29T10:53:55.206 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.223+0000 7f0bb6ffd640 10 client.4319.objecter ms_handle_connect 0x7f0ba806dd10 2024-03-29T10:53:55.210 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.228+0000 7f0bb6ffd640 1 -- 172.21.15.112:0/281848509 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0bbc050150 con 0x7f0bc00bfef0 2024-03-29T10:53:55.210 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.228+0000 7f0bb6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:55.210 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.228+0000 7f0bb6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:55.542 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.560+0000 7f0bccbcd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:55.542 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.560+0000 7f0bccbcd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:55.542 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.560+0000 7f0bccbcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:53:55.542 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.560+0000 7f0bccbcd640 1 -- 172.21.15.112:0/281848509 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0bac005330 con 0x7f0bc00bfef0 2024-03-29T10:53:55.543 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.560+0000 7f0bb6ffd640 1 -- 172.21.15.112:0/281848509 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f0bbc052250 con 0x7f0bc00bfef0 2024-03-29T10:53:55.543 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.560+0000 7f0bb6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:55.543 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.560+0000 7f0bb6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:55.544 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:53:55.545 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:55.546 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.564+0000 7f0bccbcd640 10 librados: watch_flush enter 2024-03-29T10:53:55.546 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.564+0000 7f0bccbcd640 10 librados: watch_flush exit 2024-03-29T10:53:55.546 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.564+0000 7f0bccbcd640 20 client.4319.objecter shutdown clearing up homeless session... 2024-03-29T10:53:55.546 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.564+0000 7f0bccbcd640 10 client.4319.objecter successfully canceled tick 2024-03-29T10:53:55.549 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.566+0000 7f0bccbcd640 1 -- 172.21.15.112:0/281848509 >> v1:172.21.15.118:6816/3181073408 conn(0x7f0ba806dd10 legacy=0x7f0ba80701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:55.549 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.566+0000 7f0bccbcd640 10 monclient: shutdown 2024-03-29T10:53:55.549 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.566+0000 7f0bccbcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:55.549 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.566+0000 7f0bccbcd640 1 -- 172.21.15.112:0/281848509 >> v1:172.21.15.112:6790/0 conn(0x7f0bc00bfef0 legacy=0x7f0bc0158450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:55.549 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.566+0000 7f0bccbcd640 1 -- 172.21.15.112:0/281848509 shutdown_connections 2024-03-29T10:53:55.549 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.566+0000 7f0bccbcd640 1 -- 172.21.15.112:0/281848509 >> 172.21.15.112:0/281848509 conn(0x7f0bc00058f0 msgr2=0x7f0bc014eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:55.550 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.568+0000 7f0bccbcd640 1 -- 172.21.15.112:0/281848509 shutdown_connections 2024-03-29T10:53:55.550 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.568+0000 7f0bccbcd640 1 -- 172.21.15.112:0/281848509 wait complete. 2024-03-29T10:53:55.550 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:53:55.568+0000 7f0bccbcd640 1 librados: shutdown 2024-03-29T10:53:55.571 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:02.379 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:02.535 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.553+0000 7f60299e2640 10 monclient: get_monmap_and_config 2024-03-29T10:54:02.535 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.553+0000 7f60299e2640 10 monclient: build_initial_monmap 2024-03-29T10:54:02.535 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.553+0000 7f60299e2640 10 monclient: monmap: 2024-03-29T10:54:02.535 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:54:02.535 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:02.535 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:54:02.554736+0000 2024-03-29T10:54:02.535 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:54:02.554736+0000 2024-03-29T10:54:02.535 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:02.535 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:02.535 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:54:02.535 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:54:02.536 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:54:02.536 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:02.536 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.554+0000 7f60299e2640 1 Processor -- start 2024-03-29T10:54:02.536 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.554+0000 7f60299e2640 1 -- start start 2024-03-29T10:54:02.536 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.554+0000 7f60299e2640 10 monclient: init 2024-03-29T10:54:02.537 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.555+0000 7f60299e2640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:02.537 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.555+0000 7f60299e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:02.540 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.556+0000 7f60299e2640 10 monclient(hunting): picked mon.noname-b con 0x7f60241457f0 addr v1:172.21.15.112:6790/0 2024-03-29T10:54:02.542 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.561+0000 7f60299e2640 10 monclient(hunting): picked mon.noname-a con 0x7f602413f650 addr v1:172.21.15.112:6789/0 2024-03-29T10:54:02.542 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.561+0000 7f60299e2640 10 monclient(hunting): picked mon.noname-c con 0x7f60240a9d70 addr v1:172.21.15.118:6789/0 2024-03-29T10:54:02.542 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.561+0000 7f60299e2640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60240aaef0 con 0x7f602413f650 2024-03-29T10:54:02.542 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.561+0000 7f60299e2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6024145bd0 con 0x7f60240a9d70 2024-03-29T10:54:02.542 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.561+0000 7f60299e2640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6024145eb0 con 0x7f60241457f0 2024-03-29T10:54:02.542 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.561+0000 7f60299e2640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:02.543 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.561+0000 7f60299e2640 10 monclient(hunting): authenticate will time out at 997.865547s 2024-03-29T10:54:02.543 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.561+0000 7f6022ffd640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f60241457f0 0x7f602413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:54163/0 (socket says 172.21.15.112:54163) 2024-03-29T10:54:02.543 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.561+0000 7f6022ffd640 1 -- 172.21.15.112:0/3695440670 learned_addr learned my addr 172.21.15.112:0/3695440670 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:54:02.547 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.561+0000 7f6021ffb640 1 -- 172.21.15.112:0/3695440670 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419939370 0 0) 0x7f6024145eb0 con 0x7f60241457f0 2024-03-29T10:54:02.547 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.562+0000 7f6021ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:02.547 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.562+0000 7f6021ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:02.547 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.562+0000 7f6021ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:02.547 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.562+0000 7f6021ffb640 10 monclient(hunting): my global_id is 4322 2024-03-29T10:54:02.548 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.562+0000 7f6021ffb640 1 -- 172.21.15.112:0/3695440670 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f600c003520 con 0x7f60241457f0 2024-03-29T10:54:02.548 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.562+0000 7f6021ffb640 1 -- 172.21.15.112:0/3695440670 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 142323968 0 0) 0x7f600c003520 con 0x7f60241457f0 2024-03-29T10:54:02.548 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.562+0000 7f6021ffb640 1 -- 172.21.15.112:0/3695440670 >> v1:172.21.15.118:6789/0 conn(0x7f60240a9d70 legacy=0x7f602414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:02.548 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.562+0000 7f6021ffb640 1 -- 172.21.15.112:0/3695440670 >> v1:172.21.15.112:6789/0 conn(0x7f602413f650 legacy=0x7f60240b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:02.548 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.563+0000 7f6021ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:54:02.548 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.563+0000 7f6021ffb640 1 monclient: found mon.noname-b 2024-03-29T10:54:02.549 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.563+0000 7f6021ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:02.549 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.563+0000 7f6021ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:54:02.549 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.563+0000 7f6021ffb640 1 -- 172.21.15.112:0/3695440670 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6024146340 con 0x7f60241457f0 2024-03-29T10:54:02.549 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.563+0000 7f6021ffb640 10 monclient: _finish_auth 0 2024-03-29T10:54:02.549 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.563+0000 7f6021ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:54:02.549 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.563+0000 7f6021ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:02.550 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.563+0000 7f6021ffb640 1 -- 172.21.15.112:0/3695440670 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f6018001690 con 0x7f60241457f0 2024-03-29T10:54:02.550 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.563+0000 7f60299e2640 5 monclient: authenticate success, global_id 4322 2024-03-29T10:54:02.550 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.563+0000 7f60299e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:02.551 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.563+0000 7f6021ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:02.551 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.563+0000 7f6021ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:02.551 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.563+0000 7f6021ffb640 10 monclient: dump: 2024-03-29T10:54:02.551 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:02.551 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:02.551 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:02.551 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:02.551 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:02.551 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:02.551 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:02.551 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:02.552 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:02.552 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:02.552 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.563+0000 7f60299e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:02.552 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.564+0000 7f6021ffb640 1 -- 172.21.15.112:0/3695440670 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6018001920 con 0x7f60241457f0 2024-03-29T10:54:02.552 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.565+0000 7f6021ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:02.552 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.565+0000 7f60299e2640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:02.552 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.565+0000 7f60299e2640 4 set_mon_vals no callback set 2024-03-29T10:54:02.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.565+0000 7f6021ffb640 1 -- 172.21.15.112:0/3695440670 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f6018001cb0 con 0x7f60241457f0 2024-03-29T10:54:02.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.567+0000 7f6021ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:02.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.567+0000 7f6021ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:02.559 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.567+0000 7f6021ffb640 10 monclient: dump: 2024-03-29T10:54:02.559 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:02.559 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:02.560 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:02.560 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:02.561 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:02.562 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:02.563 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:02.563 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:02.563 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:02.563 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:02.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.568+0000 7f60299e2640 10 monclient: shutdown 2024-03-29T10:54:02.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.568+0000 7f60299e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:02.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.568+0000 7f60299e2640 1 -- 172.21.15.112:0/3695440670 >> v1:172.21.15.112:6790/0 conn(0x7f60241457f0 legacy=0x7f602413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:02.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.570+0000 7f60299e2640 1 -- 172.21.15.112:0/3695440670 shutdown_connections 2024-03-29T10:54:02.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.570+0000 7f60299e2640 1 -- 172.21.15.112:0/3695440670 >> 172.21.15.112:0/3695440670 conn(0x7f6024066a00 msgr2=0x7f6024066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:02.565 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.572+0000 7f60299e2640 1 -- 172.21.15.112:0/3695440670 shutdown_connections 2024-03-29T10:54:02.565 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.572+0000 7f60299e2640 1 -- 172.21.15.112:0/3695440670 wait complete. 2024-03-29T10:54:02.566 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.573+0000 7f60299e2640 10 monclient: build_initial_monmap 2024-03-29T10:54:02.566 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.573+0000 7f60299e2640 10 monclient: monmap: 2024-03-29T10:54:02.566 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:54:02.566 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:02.567 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:54:02.567 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:54:02.567 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:02.568 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:02.568 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:54:02.569 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:54:02.570 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:54:02.571 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:02.571 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.573+0000 7f60299e2640 1 librados: starting msgr at 2024-03-29T10:54:02.572 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.573+0000 7f60299e2640 1 librados: starting objecter 2024-03-29T10:54:02.572 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.574+0000 7f60299e2640 1 Processor -- start 2024-03-29T10:54:02.577 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.574+0000 7f60299e2640 1 -- start start 2024-03-29T10:54:02.577 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.574+0000 7f60299e2640 1 librados: setting wanted keys 2024-03-29T10:54:02.577 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.574+0000 7f60299e2640 1 librados: calling monclient init 2024-03-29T10:54:02.577 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.574+0000 7f60299e2640 10 monclient: init 2024-03-29T10:54:02.577 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.575+0000 7f60299e2640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:02.578 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.575+0000 7f60299e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:02.578 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.575+0000 7f60299e2640 10 monclient(hunting): picked mon.noname-c con 0x7f60240a9d70 addr v1:172.21.15.112:6790/0 2024-03-29T10:54:02.587 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.578+0000 7f60299e2640 10 monclient(hunting): picked mon.noname-a con 0x7f602413f650 addr v1:172.21.15.112:6789/0 2024-03-29T10:54:02.588 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.578+0000 7f60299e2640 10 monclient(hunting): picked mon.noname-b con 0x7f60241457f0 addr v1:172.21.15.118:6789/0 2024-03-29T10:54:02.588 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.578+0000 7f60299e2640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f602413a3d0 con 0x7f602413f650 2024-03-29T10:54:02.588 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.578+0000 7f60299e2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6024139de0 con 0x7f60241457f0 2024-03-29T10:54:02.588 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.578+0000 7f60299e2640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6024139f70 con 0x7f60240a9d70 2024-03-29T10:54:02.588 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.578+0000 7f60299e2640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:02.588 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.578+0000 7f60299e2640 10 monclient(hunting): authenticate will time out at 997.882391s 2024-03-29T10:54:02.588 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.580+0000 7f6022ffd640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f60240a9d70 0x7f602413a8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:54279/0 (socket says 172.21.15.112:54279) 2024-03-29T10:54:02.588 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.580+0000 7f6022ffd640 1 -- 172.21.15.112:0/1710086147 learned_addr learned my addr 172.21.15.112:0/1710086147 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:54:02.588 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.581+0000 7f600bfff640 10 client.?.objecter ms_handle_connect 0x7f60240a9d70 2024-03-29T10:54:02.588 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.581+0000 7f600bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:02.588 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.582+0000 7f600bfff640 10 client.?.objecter ms_handle_connect 0x7f60241457f0 2024-03-29T10:54:02.588 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.583+0000 7f600bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:02.588 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.583+0000 7f600bfff640 1 -- 172.21.15.112:0/1710086147 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 195350251 0 0) 0x7f6024139de0 con 0x7f60241457f0 2024-03-29T10:54:02.589 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.583+0000 7f600bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:02.589 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.583+0000 7f600bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:02.589 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.583+0000 7f600bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:02.589 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.583+0000 7f600bfff640 10 monclient(hunting): my global_id is 4321 2024-03-29T10:54:02.589 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.583+0000 7f600bfff640 1 -- 172.21.15.112:0/1710086147 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6010004eb0 con 0x7f60241457f0 2024-03-29T10:54:02.589 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.583+0000 7f600bfff640 1 -- 172.21.15.112:0/1710086147 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071166639 0 0) 0x7f6024139f70 con 0x7f60240a9d70 2024-03-29T10:54:02.589 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.583+0000 7f600bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:02.589 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.583+0000 7f600bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:02.590 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.583+0000 7f600bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:02.590 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.583+0000 7f600bfff640 10 monclient(hunting): my global_id is 4325 2024-03-29T10:54:02.590 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.583+0000 7f600bfff640 1 -- 172.21.15.112:0/1710086147 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6024139de0 con 0x7f60240a9d70 2024-03-29T10:54:02.590 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.585+0000 7f600bfff640 10 client.?.objecter ms_handle_connect 0x7f602413f650 2024-03-29T10:54:02.590 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.585+0000 7f600bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:02.590 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.585+0000 7f600bfff640 1 -- 172.21.15.112:0/1710086147 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1101508758 0 0) 0x7f6010004eb0 con 0x7f60241457f0 2024-03-29T10:54:02.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.586+0000 7f600bfff640 1 -- 172.21.15.112:0/1710086147 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6024139f70 con 0x7f60241457f0 2024-03-29T10:54:02.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.586+0000 7f600bfff640 1 -- 172.21.15.112:0/1710086147 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f601c001b90 con 0x7f60241457f0 2024-03-29T10:54:02.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.586+0000 7f600bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:02.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.586+0000 7f600bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:02.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.586+0000 7f600bfff640 10 monclient(hunting): dump: 2024-03-29T10:54:02.591 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:02.592 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:02.592 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:02.592 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:02.592 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:02.592 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:02.592 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:02.592 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:02.592 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:02.593 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:02.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.586+0000 7f600bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:02.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.586+0000 7f600bfff640 1 -- 172.21.15.112:0/1710086147 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3790246722 0 0) 0x7f6024139de0 con 0x7f60240a9d70 2024-03-29T10:54:02.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.586+0000 7f600bfff640 1 -- 172.21.15.112:0/1710086147 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6010004eb0 con 0x7f60240a9d70 2024-03-29T10:54:02.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.586+0000 7f600bfff640 1 -- 172.21.15.112:0/1710086147 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481522359 0 0) 0x7f602413a3d0 con 0x7f602413f650 2024-03-29T10:54:02.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.586+0000 7f600bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:02.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.586+0000 7f600bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:02.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.586+0000 7f600bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:02.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.586+0000 7f600bfff640 10 monclient(hunting): my global_id is 4320 2024-03-29T10:54:02.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.586+0000 7f600bfff640 1 -- 172.21.15.112:0/1710086147 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6024139de0 con 0x7f602413f650 2024-03-29T10:54:02.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.587+0000 7f600bfff640 1 -- 172.21.15.112:0/1710086147 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f6018001550 con 0x7f60240a9d70 2024-03-29T10:54:02.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.587+0000 7f600bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:02.593 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.587+0000 7f600bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:02.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.587+0000 7f600bfff640 10 monclient(hunting): dump: 2024-03-29T10:54:02.594 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:02.594 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:02.594 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:02.594 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:02.594 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:02.594 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:02.594 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:02.594 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:02.594 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:02.594 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:02.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.587+0000 7f600bfff640 1 -- 172.21.15.112:0/1710086147 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3543644128 0 0) 0x7f6010004eb0 con 0x7f60240a9d70 2024-03-29T10:54:02.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.587+0000 7f600bfff640 1 -- 172.21.15.112:0/1710086147 >> v1:172.21.15.118:6789/0 conn(0x7f60241457f0 legacy=0x7f602413c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:02.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.587+0000 7f600bfff640 1 -- 172.21.15.112:0/1710086147 >> v1:172.21.15.112:6789/0 conn(0x7f602413f650 legacy=0x7f60241da5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:02.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.587+0000 7f600bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:54:02.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.587+0000 7f600bfff640 1 monclient: found mon.c 2024-03-29T10:54:02.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.587+0000 7f600bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:02.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.587+0000 7f600bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:54:02.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.587+0000 7f600bfff640 1 -- 172.21.15.112:0/1710086147 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6024142380 con 0x7f60240a9d70 2024-03-29T10:54:02.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.587+0000 7f600bfff640 10 monclient: _finish_auth 0 2024-03-29T10:54:02.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.587+0000 7f600bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:54:02.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.587+0000 7f600bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:02.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.587+0000 7f60299e2640 5 monclient: authenticate success, global_id 4325 2024-03-29T10:54:02.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.587+0000 7f60299e2640 10 monclient: _renew_subs 2024-03-29T10:54:02.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.587+0000 7f60299e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:54:02.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.587+0000 7f60299e2640 1 -- 172.21.15.112:0/1710086147 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60241425d0 con 0x7f60240a9d70 2024-03-29T10:54:02.597 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.587+0000 7f60299e2640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:02.597 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.587+0000 7f60299e2640 10 monclient: _renew_subs 2024-03-29T10:54:02.597 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.587+0000 7f60299e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:54:02.597 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.587+0000 7f60299e2640 1 -- 172.21.15.112:0/1710086147 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6024135f20 con 0x7f60240a9d70 2024-03-29T10:54:02.597 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.587+0000 7f60299e2640 1 librados: init done 2024-03-29T10:54:02.597 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.589+0000 7f60299e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:02.597 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.589+0000 7f60299e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:02.597 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.589+0000 7f60299e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:54:02.597 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.589+0000 7f60299e2640 1 -- 172.21.15.112:0/1710086147 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6024075f10 con 0x7f60240a9d70 2024-03-29T10:54:02.597 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.590+0000 7f600bfff640 1 -- 172.21.15.112:0/1710086147 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6018004a90 con 0x7f60240a9d70 2024-03-29T10:54:02.597 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.590+0000 7f600bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:02.597 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.590+0000 7f600bfff640 1 -- 172.21.15.112:0/1710086147 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f60180016b0 con 0x7f60240a9d70 2024-03-29T10:54:02.598 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.590+0000 7f600bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:02.598 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.590+0000 7f600bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:02.598 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.590+0000 7f600bfff640 10 monclient: dump: 2024-03-29T10:54:02.598 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:02.598 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:02.598 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:02.598 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:02.598 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:02.598 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:02.598 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:02.598 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:02.598 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:02.599 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:02.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.590+0000 7f600bfff640 1 -- 172.21.15.112:0/1710086147 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f601801ac10 con 0x7f60240a9d70 2024-03-29T10:54:02.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.590+0000 7f60227fc640 10 monclient: get_auth_request con 0x7f601006de00 auth_method 0 2024-03-29T10:54:02.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.590+0000 7f600bfff640 1 -- 172.21.15.112:0/1710086147 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f6018084b10 con 0x7f60240a9d70 2024-03-29T10:54:02.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.590+0000 7f600bfff640 10 client.4325.objecter ms_dispatch 0x7f6024000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:02.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.590+0000 7f600bfff640 3 client.4325.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:02.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.590+0000 7f600bfff640 3 client.4325.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:02.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.590+0000 7f600bfff640 20 client.4325.objecter dump_active .. 0 homeless 2024-03-29T10:54:02.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.591+0000 7f600bfff640 10 client.4325.objecter ms_handle_connect 0x7f601006de00 2024-03-29T10:54:02.600 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.591+0000 7f6020ff9640 4 set_mon_vals no callback set 2024-03-29T10:54:02.600 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.599+0000 7f600bfff640 1 -- 172.21.15.112:0/1710086147 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6018050470 con 0x7f60240a9d70 2024-03-29T10:54:02.600 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.599+0000 7f600bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:02.600 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.599+0000 7f600bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:02.911 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.929+0000 7f60299e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:02.912 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.929+0000 7f60299e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:02.912 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.929+0000 7f60299e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:54:02.912 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.929+0000 7f60299e2640 1 -- 172.21.15.112:0/1710086147 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60240b3680 con 0x7f60240a9d70 2024-03-29T10:54:02.912 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.930+0000 7f600bfff640 1 -- 172.21.15.112:0/1710086147 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f60180523a0 con 0x7f60240a9d70 2024-03-29T10:54:02.912 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.930+0000 7f600bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:02.912 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.930+0000 7f600bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:02.912 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:54:02.913 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:02.917 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.934+0000 7f60299e2640 10 librados: watch_flush enter 2024-03-29T10:54:02.917 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.934+0000 7f60299e2640 10 librados: watch_flush exit 2024-03-29T10:54:02.917 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.934+0000 7f60299e2640 20 client.4325.objecter shutdown clearing up homeless session... 2024-03-29T10:54:02.917 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.934+0000 7f60299e2640 10 client.4325.objecter successfully canceled tick 2024-03-29T10:54:02.917 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.934+0000 7f60299e2640 1 -- 172.21.15.112:0/1710086147 >> v1:172.21.15.118:6816/3181073408 conn(0x7f601006de00 legacy=0x7f60100702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:02.917 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.934+0000 7f60299e2640 10 monclient: shutdown 2024-03-29T10:54:02.917 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.934+0000 7f60299e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:02.917 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.934+0000 7f60299e2640 1 -- 172.21.15.112:0/1710086147 >> v1:172.21.15.112:6790/0 conn(0x7f60240a9d70 legacy=0x7f602413a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:02.917 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.935+0000 7f60299e2640 1 -- 172.21.15.112:0/1710086147 shutdown_connections 2024-03-29T10:54:02.917 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.935+0000 7f60299e2640 1 -- 172.21.15.112:0/1710086147 >> 172.21.15.112:0/1710086147 conn(0x7f6024066a00 msgr2=0x7f60240b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:02.918 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.935+0000 7f60299e2640 1 -- 172.21.15.112:0/1710086147 shutdown_connections 2024-03-29T10:54:02.918 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.935+0000 7f60299e2640 1 -- 172.21.15.112:0/1710086147 wait complete. 2024-03-29T10:54:02.918 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:02.935+0000 7f60299e2640 1 librados: shutdown 2024-03-29T10:54:02.935 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:09.742 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:09.892 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.911+0000 7f428ca9a640 10 monclient: get_monmap_and_config 2024-03-29T10:54:09.892 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.911+0000 7f428ca9a640 10 monclient: build_initial_monmap 2024-03-29T10:54:09.892 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.911+0000 7f428ca9a640 10 monclient: monmap: 2024-03-29T10:54:09.893 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:54:09.893 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:09.893 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:54:09.912260+0000 2024-03-29T10:54:09.893 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:54:09.912260+0000 2024-03-29T10:54:09.893 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:09.893 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:09.893 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:54:09.893 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:54:09.893 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:54:09.893 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:09.895 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.912+0000 7f428ca9a640 1 Processor -- start 2024-03-29T10:54:09.895 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.913+0000 7f428ca9a640 1 -- start start 2024-03-29T10:54:09.895 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.913+0000 7f428ca9a640 10 monclient: init 2024-03-29T10:54:09.895 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.914+0000 7f428ca9a640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:09.895 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.914+0000 7f428ca9a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:54:09.895 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.914+0000 7f428ca9a640 10 monclient(hunting): picked mon.noname-a con 0x7f42880b4790 addr v1:172.21.15.112:6789/0 2024-03-29T10:54:09.895 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.914+0000 7f428ca9a640 10 monclient(hunting): picked mon.noname-b con 0x7f42880aea90 addr v1:172.21.15.112:6790/0 2024-03-29T10:54:09.895 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.914+0000 7f428ca9a640 10 monclient(hunting): picked mon.noname-c con 0x7f42880b39f0 addr v1:172.21.15.118:6789/0 2024-03-29T10:54:09.896 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.914+0000 7f428ca9a640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42880b0c60 con 0x7f42880b4790 2024-03-29T10:54:09.896 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.914+0000 7f428ca9a640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42880af020 con 0x7f42880b39f0 2024-03-29T10:54:09.896 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.914+0000 7f428ca9a640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f428813fbd0 con 0x7f42880aea90 2024-03-29T10:54:09.896 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.914+0000 7f428ca9a640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:09.896 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.914+0000 7f428ca9a640 10 monclient(hunting): authenticate will time out at 1005.218674s 2024-03-29T10:54:09.898 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.917+0000 7f4285d9b640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f42880aea90 0x7f42880af540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:43353/0 (socket says 172.21.15.112:43353) 2024-03-29T10:54:09.898 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.917+0000 7f4285d9b640 1 -- 172.21.15.112:0/3181840730 learned_addr learned my addr 172.21.15.112:0/3181840730 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:54:09.899 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.918+0000 7f428559a640 1 -- 172.21.15.112:0/3181840730 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247371501 0 0) 0x7f428813fbd0 con 0x7f42880aea90 2024-03-29T10:54:09.899 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.918+0000 7f428559a640 10 monclient(hunting): _negotiate 2024-03-29T10:54:09.899 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.918+0000 7f428559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:09.899 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.918+0000 7f428559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:09.899 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.918+0000 7f428559a640 10 monclient(hunting): my global_id is 4328 2024-03-29T10:54:09.899 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.918+0000 7f428559a640 1 -- 172.21.15.112:0/3181840730 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4270003520 con 0x7f42880aea90 2024-03-29T10:54:09.900 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.919+0000 7f428559a640 1 -- 172.21.15.112:0/3181840730 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 709728839 0 0) 0x7f4270003520 con 0x7f42880aea90 2024-03-29T10:54:09.900 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.919+0000 7f428559a640 1 -- 172.21.15.112:0/3181840730 >> v1:172.21.15.118:6789/0 conn(0x7f42880b39f0 legacy=0x7f428813f4c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:54:09.902 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.919+0000 7f428559a640 1 -- 172.21.15.112:0/3181840730 >> v1:172.21.15.112:6789/0 conn(0x7f42880b4790 legacy=0x7f42880b1080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:09.902 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.919+0000 7f428559a640 10 monclient: _finish_hunting 0 2024-03-29T10:54:09.902 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.919+0000 7f428559a640 1 monclient: found mon.noname-b 2024-03-29T10:54:09.902 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.919+0000 7f428559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:09.902 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.919+0000 7f428559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:54:09.902 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.919+0000 7f428559a640 1 -- 172.21.15.112:0/3181840730 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f428813fd60 con 0x7f42880aea90 2024-03-29T10:54:09.902 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.919+0000 7f428559a640 10 monclient: _finish_auth 0 2024-03-29T10:54:09.902 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.919+0000 7f428559a640 10 monclient: _check_auth_tickets 2024-03-29T10:54:09.902 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.919+0000 7f428559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:09.902 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.919+0000 7f428559a640 1 -- 172.21.15.112:0/3181840730 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f427c001690 con 0x7f42880aea90 2024-03-29T10:54:09.902 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.919+0000 7f428559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:09.903 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.920+0000 7f428559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:09.903 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.920+0000 7f428559a640 10 monclient: dump: 2024-03-29T10:54:09.903 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:09.903 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:09.903 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:09.903 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:09.903 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:09.903 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:09.903 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:09.903 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:09.903 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:09.903 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:09.903 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.920+0000 7f428ca9a640 5 monclient: authenticate success, global_id 4328 2024-03-29T10:54:09.903 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.920+0000 7f428ca9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:09.905 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.922+0000 7f428559a640 1 -- 172.21.15.112:0/3181840730 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f427c001860 con 0x7f42880aea90 2024-03-29T10:54:09.905 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.922+0000 7f428559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:09.905 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.922+0000 7f428559a640 1 -- 172.21.15.112:0/3181840730 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f427c001c70 con 0x7f42880aea90 2024-03-29T10:54:09.905 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.922+0000 7f428559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:09.905 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.922+0000 7f428559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:09.905 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.922+0000 7f428559a640 10 monclient: dump: 2024-03-29T10:54:09.905 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:09.905 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:09.905 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:09.905 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:09.905 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:09.905 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:09.905 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:09.906 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:09.906 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:09.906 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:09.906 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.922+0000 7f428ca9a640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:09.906 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.922+0000 7f428ca9a640 4 set_mon_vals no callback set 2024-03-29T10:54:09.906 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.923+0000 7f428ca9a640 10 monclient: shutdown 2024-03-29T10:54:09.906 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.923+0000 7f428ca9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:09.906 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.923+0000 7f428ca9a640 1 -- 172.21.15.112:0/3181840730 >> v1:172.21.15.112:6790/0 conn(0x7f42880aea90 legacy=0x7f42880af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:09.907 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.924+0000 7f428ca9a640 1 -- 172.21.15.112:0/3181840730 shutdown_connections 2024-03-29T10:54:09.907 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.924+0000 7f428ca9a640 1 -- 172.21.15.112:0/3181840730 >> 172.21.15.112:0/3181840730 conn(0x7f4288066a00 msgr2=0x7f4288066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:09.909 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.928+0000 7f428ca9a640 1 -- 172.21.15.112:0/3181840730 shutdown_connections 2024-03-29T10:54:09.912 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.929+0000 7f428ca9a640 1 -- 172.21.15.112:0/3181840730 wait complete. 2024-03-29T10:54:09.912 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.930+0000 7f428ca9a640 10 monclient: build_initial_monmap 2024-03-29T10:54:09.912 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.930+0000 7f428ca9a640 10 monclient: monmap: 2024-03-29T10:54:09.912 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:54:09.913 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:09.914 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:54:09.914 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:54:09.914 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:09.914 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:09.914 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:54:09.914 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:54:09.914 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:54:09.914 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:09.914 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.931+0000 7f428ca9a640 1 librados: starting msgr at 2024-03-29T10:54:09.914 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.931+0000 7f428ca9a640 1 librados: starting objecter 2024-03-29T10:54:09.914 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.931+0000 7f428ca9a640 1 Processor -- start 2024-03-29T10:54:09.916 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.932+0000 7f428ca9a640 1 -- start start 2024-03-29T10:54:09.916 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.932+0000 7f428ca9a640 1 librados: setting wanted keys 2024-03-29T10:54:09.916 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.932+0000 7f428ca9a640 1 librados: calling monclient init 2024-03-29T10:54:09.916 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.932+0000 7f428ca9a640 10 monclient: init 2024-03-29T10:54:09.916 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.933+0000 7f428ca9a640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:09.916 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.933+0000 7f428ca9a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:54:09.916 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.933+0000 7f428ca9a640 10 monclient(hunting): picked mon.noname-a con 0x7f42880b39f0 addr v1:172.21.15.112:6789/0 2024-03-29T10:54:09.916 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.933+0000 7f428ca9a640 10 monclient(hunting): picked mon.noname-c con 0x7f42880b4790 addr v1:172.21.15.112:6790/0 2024-03-29T10:54:09.916 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.933+0000 7f428ca9a640 10 monclient(hunting): picked mon.noname-b con 0x7f4288141cc0 addr v1:172.21.15.118:6789/0 2024-03-29T10:54:09.917 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.933+0000 7f428ca9a640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42880acb40 con 0x7f42880b39f0 2024-03-29T10:54:09.917 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.933+0000 7f428ca9a640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42880ace20 con 0x7f4288141cc0 2024-03-29T10:54:09.917 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.933+0000 7f428ca9a640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42880ab380 con 0x7f42880b4790 2024-03-29T10:54:09.917 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.933+0000 7f428ca9a640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:09.917 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.933+0000 7f428ca9a640 10 monclient(hunting): authenticate will time out at 1005.237694s 2024-03-29T10:54:09.922 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.941+0000 7f4286d9d640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f4288141cc0 0x7f42881e2090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:46741/0 (socket says 172.21.15.112:46741) 2024-03-29T10:54:09.922 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.942+0000 7f4286d9d640 1 -- 172.21.15.112:0/3598522190 learned_addr learned my addr 172.21.15.112:0/3598522190 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:54:09.924 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.943+0000 7f4276ffd640 10 client.?.objecter ms_handle_connect 0x7f42880b4790 2024-03-29T10:54:09.924 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.943+0000 7f4276ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:09.925 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.944+0000 7f4276ffd640 1 -- 172.21.15.112:0/3598522190 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878763 0 0) 0x7f42880ab380 con 0x7f42880b4790 2024-03-29T10:54:09.925 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.944+0000 7f4276ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:09.926 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.944+0000 7f4276ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:09.926 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.944+0000 7f4276ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:09.926 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.944+0000 7f4276ffd640 10 monclient(hunting): my global_id is 4331 2024-03-29T10:54:09.926 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.944+0000 7f4276ffd640 1 -- 172.21.15.112:0/3598522190 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4268004eb0 con 0x7f42880b4790 2024-03-29T10:54:09.926 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.945+0000 7f4276ffd640 10 client.?.objecter ms_handle_connect 0x7f42880b39f0 2024-03-29T10:54:09.926 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.945+0000 7f4276ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:09.931 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.945+0000 7f4276ffd640 1 -- 172.21.15.112:0/3598522190 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1264334326 0 0) 0x7f42880acb40 con 0x7f42880b39f0 2024-03-29T10:54:09.931 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.945+0000 7f4276ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:09.931 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.945+0000 7f4276ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:09.931 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.945+0000 7f4276ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:09.932 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.945+0000 7f4276ffd640 10 monclient(hunting): my global_id is 4326 2024-03-29T10:54:09.932 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.945+0000 7f4276ffd640 1 -- 172.21.15.112:0/3598522190 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42880ab380 con 0x7f42880b39f0 2024-03-29T10:54:09.932 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.945+0000 7f4276ffd640 10 client.?.objecter ms_handle_connect 0x7f4288141cc0 2024-03-29T10:54:09.932 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.946+0000 7f4276ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:09.932 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.946+0000 7f4276ffd640 1 -- 172.21.15.112:0/3598522190 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 442202308 0 0) 0x7f42880ace20 con 0x7f4288141cc0 2024-03-29T10:54:09.932 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.946+0000 7f4276ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:09.932 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.946+0000 7f4276ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:09.932 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.946+0000 7f4276ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:09.932 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.946+0000 7f4276ffd640 10 monclient(hunting): my global_id is 4324 2024-03-29T10:54:09.932 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.946+0000 7f4276ffd640 1 -- 172.21.15.112:0/3598522190 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42880acb40 con 0x7f4288141cc0 2024-03-29T10:54:09.932 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.946+0000 7f4276ffd640 1 -- 172.21.15.112:0/3598522190 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2347145124 0 0) 0x7f42880acb40 con 0x7f4288141cc0 2024-03-29T10:54:09.932 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.946+0000 7f4276ffd640 1 -- 172.21.15.112:0/3598522190 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42880ace20 con 0x7f4288141cc0 2024-03-29T10:54:09.932 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.947+0000 7f4276ffd640 1 -- 172.21.15.112:0/3598522190 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f4280004280 con 0x7f4288141cc0 2024-03-29T10:54:09.932 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.947+0000 7f4276ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:09.933 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.947+0000 7f4276ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:09.933 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.947+0000 7f4276ffd640 10 monclient(hunting): dump: 2024-03-29T10:54:09.933 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:09.933 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:09.933 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:09.933 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:09.933 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:09.933 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:09.933 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:09.933 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:09.933 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:09.933 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:09.933 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.947+0000 7f4276ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:09.933 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.947+0000 7f4276ffd640 1 -- 172.21.15.112:0/3598522190 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1945207620 0 0) 0x7f42880ace20 con 0x7f4288141cc0 2024-03-29T10:54:09.934 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.947+0000 7f4276ffd640 1 -- 172.21.15.112:0/3598522190 >> v1:172.21.15.112:6790/0 conn(0x7f42880b4790 legacy=0x7f42880ad240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:09.934 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.947+0000 7f4276ffd640 1 -- 172.21.15.112:0/3598522190 >> v1:172.21.15.112:6789/0 conn(0x7f42880b39f0 legacy=0x7f42880ab8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:09.934 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.947+0000 7f4276ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:54:09.934 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.947+0000 7f4276ffd640 1 monclient: found mon.b 2024-03-29T10:54:09.934 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.947+0000 7f4276ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:09.934 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.947+0000 7f4276ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:54:09.934 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.947+0000 7f4276ffd640 1 -- 172.21.15.112:0/3598522190 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42880ad950 con 0x7f4288141cc0 2024-03-29T10:54:09.934 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.947+0000 7f4276ffd640 10 monclient: _finish_auth 0 2024-03-29T10:54:09.934 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.947+0000 7f4276ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:54:09.934 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.948+0000 7f4276ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:09.934 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.948+0000 7f428ca9a640 5 monclient: authenticate success, global_id 4324 2024-03-29T10:54:09.934 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.948+0000 7f428ca9a640 10 monclient: _renew_subs 2024-03-29T10:54:09.934 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.948+0000 7f428ca9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:54:09.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.948+0000 7f428ca9a640 1 -- 172.21.15.112:0/3598522190 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42880aa7c0 con 0x7f4288141cc0 2024-03-29T10:54:09.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.948+0000 7f428ca9a640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:09.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.948+0000 7f428ca9a640 10 monclient: _renew_subs 2024-03-29T10:54:09.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.948+0000 7f428ca9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:54:09.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.948+0000 7f428ca9a640 1 -- 172.21.15.112:0/3598522190 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42880ab130 con 0x7f4288141cc0 2024-03-29T10:54:09.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.948+0000 7f428ca9a640 1 librados: init done 2024-03-29T10:54:09.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.948+0000 7f4276ffd640 1 -- 172.21.15.112:0/3598522190 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4280004850 con 0x7f4288141cc0 2024-03-29T10:54:09.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.948+0000 7f4276ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:09.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.948+0000 7f4276ffd640 1 -- 172.21.15.112:0/3598522190 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f42800013f0 con 0x7f4288141cc0 2024-03-29T10:54:09.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.948+0000 7f4276ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:09.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.948+0000 7f4276ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:09.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.948+0000 7f4276ffd640 10 monclient: dump: 2024-03-29T10:54:09.936 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:09.936 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:09.936 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:09.936 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:09.936 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:09.937 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:09.937 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:09.937 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:09.937 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:09.937 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:09.937 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.949+0000 7f4277fff640 4 set_mon_vals no callback set 2024-03-29T10:54:09.937 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.949+0000 7f428ca9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:09.937 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.950+0000 7f4276ffd640 1 -- 172.21.15.112:0/3598522190 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f4280001550 con 0x7f4288141cc0 2024-03-29T10:54:09.937 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.950+0000 7f428ca9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:09.937 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.950+0000 7f428ca9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:54:09.937 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.950+0000 7f428ca9a640 1 -- 172.21.15.112:0/3598522190 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f426c005080 con 0x7f4288141cc0 2024-03-29T10:54:09.938 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.951+0000 7f4276ffd640 1 -- 172.21.15.112:0/3598522190 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f4280084c50 con 0x7f4288141cc0 2024-03-29T10:54:09.939 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.951+0000 7f4276ffd640 10 client.4324.objecter ms_dispatch 0x7f4288000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:09.939 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.951+0000 7f4276ffd640 3 client.4324.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:09.939 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.951+0000 7f4276ffd640 3 client.4324.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:09.939 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.951+0000 7f4276ffd640 20 client.4324.objecter dump_active .. 0 homeless 2024-03-29T10:54:09.939 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.952+0000 7f428659c640 10 monclient: get_auth_request con 0x7f426806de80 auth_method 0 2024-03-29T10:54:09.939 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.953+0000 7f4276ffd640 10 client.4324.objecter ms_handle_connect 0x7f426806de80 2024-03-29T10:54:09.940 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.959+0000 7f4276ffd640 1 -- 172.21.15.112:0/3598522190 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4280050530 con 0x7f4288141cc0 2024-03-29T10:54:09.940 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.959+0000 7f4276ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:09.941 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:09.960+0000 7f4276ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:10.266 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:10.284+0000 7f428ca9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:10.266 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:10.285+0000 7f428ca9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:10.266 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:10.285+0000 7f428ca9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:54:10.266 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:10.285+0000 7f428ca9a640 1 -- 172.21.15.112:0/3598522190 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f426c005330 con 0x7f4288141cc0 2024-03-29T10:54:10.269 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:10.288+0000 7f4276ffd640 1 -- 172.21.15.112:0/3598522190 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f4280052460 con 0x7f4288141cc0 2024-03-29T10:54:10.269 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:10.288+0000 7f4276ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:10.269 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:10.288+0000 7f4276ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:10.270 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:54:10.270 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:10.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:10.292+0000 7f428ca9a640 10 librados: watch_flush enter 2024-03-29T10:54:10.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:10.292+0000 7f428ca9a640 10 librados: watch_flush exit 2024-03-29T10:54:10.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:10.292+0000 7f428ca9a640 20 client.4324.objecter shutdown clearing up homeless session... 2024-03-29T10:54:10.274 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:10.292+0000 7f428ca9a640 10 client.4324.objecter successfully canceled tick 2024-03-29T10:54:10.274 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:10.292+0000 7f428ca9a640 1 -- 172.21.15.112:0/3598522190 >> v1:172.21.15.118:6816/3181073408 conn(0x7f426806de80 legacy=0x7f4268070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:10.274 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:10.292+0000 7f428ca9a640 10 monclient: shutdown 2024-03-29T10:54:10.274 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:10.292+0000 7f428ca9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:10.274 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:10.292+0000 7f428ca9a640 1 -- 172.21.15.112:0/3598522190 >> v1:172.21.15.118:6789/0 conn(0x7f4288141cc0 legacy=0x7f42881e2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:10.275 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:10.293+0000 7f428ca9a640 1 -- 172.21.15.112:0/3598522190 shutdown_connections 2024-03-29T10:54:10.275 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:10.293+0000 7f428ca9a640 1 -- 172.21.15.112:0/3598522190 >> 172.21.15.112:0/3598522190 conn(0x7f4288066a00 msgr2=0x7f42880b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:10.275 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:10.293+0000 7f428ca9a640 1 -- 172.21.15.112:0/3598522190 shutdown_connections 2024-03-29T10:54:10.275 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:10.293+0000 7f428ca9a640 1 -- 172.21.15.112:0/3598522190 wait complete. 2024-03-29T10:54:10.275 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:10.293+0000 7f428ca9a640 1 librados: shutdown 2024-03-29T10:54:10.293 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:17.100 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:17.248 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.266+0000 7fee5816d640 10 monclient: get_monmap_and_config 2024-03-29T10:54:17.248 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.266+0000 7fee5816d640 10 monclient: build_initial_monmap 2024-03-29T10:54:17.248 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.266+0000 7fee5816d640 10 monclient: monmap: 2024-03-29T10:54:17.248 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:54:17.248 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:17.248 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:54:17.267058+0000 2024-03-29T10:54:17.248 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:54:17.267058+0000 2024-03-29T10:54:17.248 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:17.249 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:17.249 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:54:17.249 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:54:17.249 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:54:17.249 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:17.249 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.268+0000 7fee5816d640 1 Processor -- start 2024-03-29T10:54:17.249 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.268+0000 7fee5816d640 1 -- start start 2024-03-29T10:54:17.249 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.268+0000 7fee5816d640 10 monclient: init 2024-03-29T10:54:17.250 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.269+0000 7fee5816d640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:17.250 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.269+0000 7fee5816d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:17.250 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.269+0000 7fee5816d640 10 monclient(hunting): picked mon.noname-a con 0x7fee500a9ac0 addr v1:172.21.15.112:6789/0 2024-03-29T10:54:17.250 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.269+0000 7fee5816d640 10 monclient(hunting): picked mon.noname-c con 0x7fee5013ee90 addr v1:172.21.15.118:6789/0 2024-03-29T10:54:17.251 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.269+0000 7fee5816d640 10 monclient(hunting): picked mon.noname-b con 0x7fee500af900 addr v1:172.21.15.112:6790/0 2024-03-29T10:54:17.252 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.270+0000 7fee5816d640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee500afd10 con 0x7fee500a9ac0 2024-03-29T10:54:17.252 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.270+0000 7fee5816d640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee500af6f0 con 0x7fee5013ee90 2024-03-29T10:54:17.252 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.270+0000 7fee5816d640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee500aac80 con 0x7fee500af900 2024-03-29T10:54:17.252 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.270+0000 7fee5816d640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:17.252 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.270+0000 7fee5816d640 10 monclient(hunting): authenticate will time out at 1012.574978s 2024-03-29T10:54:17.254 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.272+0000 7fee5670a640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fee500af900 0x7fee50149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59831/0 (socket says 172.21.15.112:59831) 2024-03-29T10:54:17.254 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.272+0000 7fee5670a640 1 -- 172.21.15.112:0/1472496515 learned_addr learned my addr 172.21.15.112:0/1472496515 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:54:17.254 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.273+0000 7fee54f07640 1 -- 172.21.15.112:0/1472496515 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249644451 0 0) 0x7fee500aac80 con 0x7fee500af900 2024-03-29T10:54:17.254 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.273+0000 7fee54f07640 10 monclient(hunting): _negotiate 2024-03-29T10:54:17.254 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.273+0000 7fee54f07640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:17.254 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.273+0000 7fee54f07640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:17.254 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.273+0000 7fee54f07640 10 monclient(hunting): my global_id is 4334 2024-03-29T10:54:17.254 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.273+0000 7fee54f07640 1 -- 172.21.15.112:0/1472496515 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee3c003520 con 0x7fee500af900 2024-03-29T10:54:17.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.273+0000 7fee54f07640 1 -- 172.21.15.112:0/1472496515 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 765956594 0 0) 0x7fee3c003520 con 0x7fee500af900 2024-03-29T10:54:17.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.273+0000 7fee54f07640 1 -- 172.21.15.112:0/1472496515 >> v1:172.21.15.118:6789/0 conn(0x7fee5013ee90 legacy=0x7fee5013b5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:54:17.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.273+0000 7fee54f07640 1 -- 172.21.15.112:0/1472496515 >> v1:172.21.15.112:6789/0 conn(0x7fee500a9ac0 legacy=0x7fee500adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:54:17.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.273+0000 7fee54f07640 10 monclient: _finish_hunting 0 2024-03-29T10:54:17.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.273+0000 7fee54f07640 1 monclient: found mon.noname-b 2024-03-29T10:54:17.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.273+0000 7fee54f07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:17.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.273+0000 7fee54f07640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:54:17.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.273+0000 7fee54f07640 1 -- 172.21.15.112:0/1472496515 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee500aaf60 con 0x7fee500af900 2024-03-29T10:54:17.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.273+0000 7fee54f07640 10 monclient: _finish_auth 0 2024-03-29T10:54:17.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.273+0000 7fee54f07640 10 monclient: _check_auth_tickets 2024-03-29T10:54:17.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.273+0000 7fee54f07640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:17.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.273+0000 7fee54f07640 1 -- 172.21.15.112:0/1472496515 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fee48001a10 con 0x7fee500af900 2024-03-29T10:54:17.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.273+0000 7fee5816d640 5 monclient: authenticate success, global_id 4334 2024-03-29T10:54:17.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.274+0000 7fee5816d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:17.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.274+0000 7fee54f07640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:17.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.274+0000 7fee54f07640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:17.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.274+0000 7fee54f07640 10 monclient: dump: 2024-03-29T10:54:17.257 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:17.258 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:17.258 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:17.258 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:17.258 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:17.258 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:17.258 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:17.258 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:17.258 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:17.258 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:17.258 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.274+0000 7fee54f07640 1 -- 172.21.15.112:0/1472496515 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee48001ca0 con 0x7fee500af900 2024-03-29T10:54:17.258 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.274+0000 7fee5816d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:17.258 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.274+0000 7fee54f07640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:17.258 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.274+0000 7fee54f07640 1 -- 172.21.15.112:0/1472496515 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fee48002030 con 0x7fee500af900 2024-03-29T10:54:17.258 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.274+0000 7fee5816d640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:17.259 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.274+0000 7fee5816d640 4 set_mon_vals no callback set 2024-03-29T10:54:17.259 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.275+0000 7fee5816d640 10 monclient: shutdown 2024-03-29T10:54:17.259 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.275+0000 7fee5816d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:17.259 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.275+0000 7fee5816d640 1 -- 172.21.15.112:0/1472496515 >> v1:172.21.15.112:6790/0 conn(0x7fee500af900 legacy=0x7fee50149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:17.259 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.275+0000 7fee54f07640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:54:17.260 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.276+0000 7fee5816d640 1 -- 172.21.15.112:0/1472496515 shutdown_connections 2024-03-29T10:54:17.260 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.276+0000 7fee5816d640 1 -- 172.21.15.112:0/1472496515 >> 172.21.15.112:0/1472496515 conn(0x7fee500b35e0 msgr2=0x7fee500b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:17.260 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.277+0000 7fee5816d640 1 -- 172.21.15.112:0/1472496515 shutdown_connections 2024-03-29T10:54:17.260 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.277+0000 7fee5816d640 1 -- 172.21.15.112:0/1472496515 wait complete. 2024-03-29T10:54:17.260 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.278+0000 7fee5816d640 10 monclient: build_initial_monmap 2024-03-29T10:54:17.260 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.278+0000 7fee5816d640 10 monclient: monmap: 2024-03-29T10:54:17.260 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:54:17.260 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:17.260 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:54:17.260 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:54:17.260 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:17.261 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:17.261 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:54:17.261 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:54:17.261 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:54:17.261 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:17.261 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.279+0000 7fee5816d640 1 librados: starting msgr at 2024-03-29T10:54:17.261 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.279+0000 7fee5816d640 1 librados: starting objecter 2024-03-29T10:54:17.261 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.279+0000 7fee5816d640 1 Processor -- start 2024-03-29T10:54:17.262 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.279+0000 7fee5816d640 1 -- start start 2024-03-29T10:54:17.262 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.279+0000 7fee5816d640 1 librados: setting wanted keys 2024-03-29T10:54:17.262 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.279+0000 7fee5816d640 1 librados: calling monclient init 2024-03-29T10:54:17.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.279+0000 7fee5816d640 10 monclient: init 2024-03-29T10:54:17.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.280+0000 7fee5816d640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:17.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.280+0000 7fee5816d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:17.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.280+0000 7fee5816d640 10 monclient(hunting): picked mon.noname-a con 0x7fee500a9ac0 addr v1:172.21.15.112:6789/0 2024-03-29T10:54:17.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.280+0000 7fee5816d640 10 monclient(hunting): picked mon.noname-b con 0x7fee500af900 addr v1:172.21.15.118:6789/0 2024-03-29T10:54:17.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.280+0000 7fee5816d640 10 monclient(hunting): picked mon.noname-c con 0x7fee5013ee90 addr v1:172.21.15.112:6790/0 2024-03-29T10:54:17.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.280+0000 7fee5816d640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee50070790 con 0x7fee500a9ac0 2024-03-29T10:54:17.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.280+0000 7fee5816d640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee501d9950 con 0x7fee500af900 2024-03-29T10:54:17.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.280+0000 7fee5816d640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee50071560 con 0x7fee5013ee90 2024-03-29T10:54:17.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.280+0000 7fee5816d640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:17.264 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.280+0000 7fee5816d640 10 monclient(hunting): authenticate will time out at 1012.585094s 2024-03-29T10:54:17.267 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.286+0000 7fee55708640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fee500af900 0x7fee501d9240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:39711/0 (socket says 172.21.15.112:39711) 2024-03-29T10:54:17.267 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.286+0000 7fee5670a640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fee5013ee90 0x7fee5006f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59885/0 (socket says 172.21.15.112:59885) 2024-03-29T10:54:17.267 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.286+0000 7fee5670a640 1 -- 172.21.15.112:0/2440460408 learned_addr learned my addr 172.21.15.112:0/2440460408 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:54:17.267 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.286+0000 7fee55f09640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fee500a9ac0 0x7fee501d8b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:39187/0 (socket says 172.21.15.112:39187) 2024-03-29T10:54:17.268 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.286+0000 7fee467fc640 10 client.?.objecter ms_handle_connect 0x7fee5013ee90 2024-03-29T10:54:17.268 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.286+0000 7fee467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:17.269 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.287+0000 7fee467fc640 10 client.?.objecter ms_handle_connect 0x7fee500a9ac0 2024-03-29T10:54:17.269 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.287+0000 7fee467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:17.270 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.287+0000 7fee467fc640 1 -- 172.21.15.112:0/2440460408 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1109945065 0 0) 0x7fee50071560 con 0x7fee5013ee90 2024-03-29T10:54:17.270 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.287+0000 7fee467fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:17.270 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.287+0000 7fee467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:17.270 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.287+0000 7fee467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:17.270 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.287+0000 7fee467fc640 10 monclient(hunting): my global_id is 4337 2024-03-29T10:54:17.270 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.287+0000 7fee467fc640 1 -- 172.21.15.112:0/2440460408 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee34004eb0 con 0x7fee5013ee90 2024-03-29T10:54:17.270 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.287+0000 7fee467fc640 10 client.?.objecter ms_handle_connect 0x7fee500af900 2024-03-29T10:54:17.270 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.287+0000 7fee467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:17.270 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.287+0000 7fee467fc640 1 -- 172.21.15.112:0/2440460408 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 20437723 0 0) 0x7fee34004eb0 con 0x7fee5013ee90 2024-03-29T10:54:17.270 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 1 -- 172.21.15.112:0/2440460408 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee34005300 con 0x7fee5013ee90 2024-03-29T10:54:17.270 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 1 -- 172.21.15.112:0/2440460408 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934546656 0 0) 0x7fee50070790 con 0x7fee500a9ac0 2024-03-29T10:54:17.270 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:17.270 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:17.271 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:17.271 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 10 monclient(hunting): my global_id is 4329 2024-03-29T10:54:17.271 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 1 -- 172.21.15.112:0/2440460408 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee34005760 con 0x7fee500a9ac0 2024-03-29T10:54:17.271 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 1 -- 172.21.15.112:0/2440460408 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fee480018d0 con 0x7fee5013ee90 2024-03-29T10:54:17.271 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:17.271 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:17.271 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 10 monclient(hunting): dump: 2024-03-29T10:54:17.271 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:17.271 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:17.271 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:17.271 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:17.271 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:17.272 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:17.272 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:17.272 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:17.272 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:17.272 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:17.272 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:17.272 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 1 -- 172.21.15.112:0/2440460408 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2709558877 0 0) 0x7fee34005300 con 0x7fee5013ee90 2024-03-29T10:54:17.272 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 1 -- 172.21.15.112:0/2440460408 >> v1:172.21.15.118:6789/0 conn(0x7fee500af900 legacy=0x7fee501d9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:17.272 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 1 -- 172.21.15.112:0/2440460408 >> v1:172.21.15.112:6789/0 conn(0x7fee500a9ac0 legacy=0x7fee501d8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:17.272 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 10 monclient: _finish_hunting 0 2024-03-29T10:54:17.272 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 1 monclient: found mon.c 2024-03-29T10:54:17.272 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:17.272 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:54:17.272 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 1 -- 172.21.15.112:0/2440460408 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee50071840 con 0x7fee5013ee90 2024-03-29T10:54:17.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 10 monclient: _finish_auth 0 2024-03-29T10:54:17.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 10 monclient: _check_auth_tickets 2024-03-29T10:54:17.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:17.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 1 -- 172.21.15.112:0/2440460408 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee48004f70 con 0x7fee5013ee90 2024-03-29T10:54:17.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:17.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 1 -- 172.21.15.112:0/2440460408 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fee480018d0 con 0x7fee5013ee90 2024-03-29T10:54:17.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:17.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:17.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.288+0000 7fee467fc640 10 monclient: dump: 2024-03-29T10:54:17.273 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:17.273 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:17.273 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:17.273 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:17.273 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:17.273 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:17.274 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:17.274 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:17.274 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:17.274 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:17.275 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.289+0000 7fee47fff640 4 set_mon_vals no callback set 2024-03-29T10:54:17.276 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.291+0000 7fee5816d640 5 monclient: authenticate success, global_id 4337 2024-03-29T10:54:17.276 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.292+0000 7fee5816d640 10 monclient: _renew_subs 2024-03-29T10:54:17.276 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.292+0000 7fee5816d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:54:17.276 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.292+0000 7fee5816d640 1 -- 172.21.15.112:0/2440460408 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee50071b20 con 0x7fee5013ee90 2024-03-29T10:54:17.276 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.292+0000 7fee5816d640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:17.276 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.292+0000 7fee5816d640 10 monclient: _renew_subs 2024-03-29T10:54:17.276 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.292+0000 7fee5816d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:54:17.276 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.292+0000 7fee5816d640 1 -- 172.21.15.112:0/2440460408 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee5006fec0 con 0x7fee5013ee90 2024-03-29T10:54:17.276 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.292+0000 7fee5816d640 1 librados: init done 2024-03-29T10:54:17.276 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.294+0000 7fee467fc640 1 -- 172.21.15.112:0/2440460408 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fee4801afd0 con 0x7fee5013ee90 2024-03-29T10:54:17.277 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.295+0000 7fee5816d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:17.277 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.295+0000 7fee5816d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:17.277 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.295+0000 7fee5816d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:54:17.277 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.295+0000 7fee5816d640 1 -- 172.21.15.112:0/2440460408 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee500aa900 con 0x7fee5013ee90 2024-03-29T10:54:17.277 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.296+0000 7fee467fc640 1 -- 172.21.15.112:0/2440460408 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7fee48084ac0 con 0x7fee5013ee90 2024-03-29T10:54:17.277 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.296+0000 7fee467fc640 10 client.4337.objecter ms_dispatch 0x7fee50000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:17.277 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.296+0000 7fee467fc640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:17.277 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.296+0000 7fee467fc640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:17.278 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.296+0000 7fee55f09640 10 monclient: get_auth_request con 0x7fee3406e540 auth_method 0 2024-03-29T10:54:17.278 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.297+0000 7fee467fc640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-29T10:54:17.278 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.297+0000 7fee467fc640 10 client.4337.objecter ms_handle_connect 0x7fee3406e540 2024-03-29T10:54:17.285 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.304+0000 7fee467fc640 1 -- 172.21.15.112:0/2440460408 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee480501d0 con 0x7fee5013ee90 2024-03-29T10:54:17.285 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.304+0000 7fee467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:17.285 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.304+0000 7fee467fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:17.600 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.619+0000 7fee5816d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:17.600 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.619+0000 7fee5816d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:17.600 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.619+0000 7fee5816d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:54:17.600 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.619+0000 7fee5816d640 1 -- 172.21.15.112:0/2440460408 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee500aa900 con 0x7fee5013ee90 2024-03-29T10:54:17.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.620+0000 7fee467fc640 1 -- 172.21.15.112:0/2440460408 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7fee480522d0 con 0x7fee5013ee90 2024-03-29T10:54:17.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.620+0000 7fee467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:17.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.620+0000 7fee467fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:17.602 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:54:17.602 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:17.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.624+0000 7fee5816d640 10 librados: watch_flush enter 2024-03-29T10:54:17.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.624+0000 7fee5816d640 10 librados: watch_flush exit 2024-03-29T10:54:17.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.624+0000 7fee5816d640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-29T10:54:17.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.624+0000 7fee5816d640 10 client.4337.objecter successfully canceled tick 2024-03-29T10:54:17.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.624+0000 7fee5816d640 1 -- 172.21.15.112:0/2440460408 >> v1:172.21.15.118:6816/3181073408 conn(0x7fee3406e540 legacy=0x7fee34070a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:17.606 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.624+0000 7fee5816d640 10 monclient: shutdown 2024-03-29T10:54:17.606 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.624+0000 7fee5816d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:17.606 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.624+0000 7fee5816d640 1 -- 172.21.15.112:0/2440460408 >> v1:172.21.15.112:6790/0 conn(0x7fee5013ee90 legacy=0x7fee5006f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:17.606 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.624+0000 7fee5816d640 1 -- 172.21.15.112:0/2440460408 shutdown_connections 2024-03-29T10:54:17.606 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.624+0000 7fee5816d640 1 -- 172.21.15.112:0/2440460408 >> 172.21.15.112:0/2440460408 conn(0x7fee500b35e0 msgr2=0x7fee5012fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:17.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.626+0000 7fee5816d640 1 -- 172.21.15.112:0/2440460408 shutdown_connections 2024-03-29T10:54:17.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.626+0000 7fee5816d640 1 -- 172.21.15.112:0/2440460408 wait complete. 2024-03-29T10:54:17.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:17.626+0000 7fee5816d640 1 librados: shutdown 2024-03-29T10:54:17.631 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:24.439 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:24.590 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.609+0000 7f3de87b5640 10 monclient: get_monmap_and_config 2024-03-29T10:54:24.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.609+0000 7f3de87b5640 10 monclient: build_initial_monmap 2024-03-29T10:54:24.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.609+0000 7f3de87b5640 10 monclient: monmap: 2024-03-29T10:54:24.591 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:54:24.591 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:24.591 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:54:24.610509+0000 2024-03-29T10:54:24.591 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:54:24.610509+0000 2024-03-29T10:54:24.591 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:24.591 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:24.591 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:54:24.591 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:54:24.591 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:54:24.591 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:24.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.611+0000 7f3de87b5640 1 Processor -- start 2024-03-29T10:54:24.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.611+0000 7f3de87b5640 1 -- start start 2024-03-29T10:54:24.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.611+0000 7f3de87b5640 10 monclient: init 2024-03-29T10:54:24.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.612+0000 7f3de87b5640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:24.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.612+0000 7f3de87b5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:54:24.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.612+0000 7f3de87b5640 10 monclient(hunting): picked mon.noname-a con 0x7f3de006d260 addr v1:172.21.15.112:6789/0 2024-03-29T10:54:24.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.612+0000 7f3de87b5640 10 monclient(hunting): picked mon.noname-b con 0x7f3de00723b0 addr v1:172.21.15.112:6790/0 2024-03-29T10:54:24.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.612+0000 7f3de87b5640 10 monclient(hunting): picked mon.noname-c con 0x7f3de0146b80 addr v1:172.21.15.118:6789/0 2024-03-29T10:54:24.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.612+0000 7f3de87b5640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de00701d0 con 0x7f3de006d260 2024-03-29T10:54:24.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.612+0000 7f3de87b5640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de0070440 con 0x7f3de0146b80 2024-03-29T10:54:24.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.612+0000 7f3de87b5640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de006c2c0 con 0x7f3de00723b0 2024-03-29T10:54:24.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.612+0000 7f3de87b5640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:24.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.612+0000 7f3de87b5640 10 monclient(hunting): authenticate will time out at 1019.916586s 2024-03-29T10:54:24.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.613+0000 7f3de6551640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f3de006d260 0x7f3de0071ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60710/0 (socket says 172.21.15.112:60710) 2024-03-29T10:54:24.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.613+0000 7f3de6551640 1 -- 172.21.15.112:0/746906765 learned_addr learned my addr 172.21.15.112:0/746906765 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:54:24.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.614+0000 7f3de554f640 1 -- 172.21.15.112:0/746906765 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 417752635 0 0) 0x7f3de00701d0 con 0x7f3de006d260 2024-03-29T10:54:24.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.614+0000 7f3de554f640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.614+0000 7f3de554f640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.614+0000 7f3de554f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.614+0000 7f3de554f640 10 monclient(hunting): my global_id is 4332 2024-03-29T10:54:24.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.614+0000 7f3de554f640 1 -- 172.21.15.112:0/746906765 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dcc003520 con 0x7f3de006d260 2024-03-29T10:54:24.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.614+0000 7f3de554f640 1 -- 172.21.15.112:0/746906765 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4275913629 0 0) 0x7f3dcc003520 con 0x7f3de006d260 2024-03-29T10:54:24.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.614+0000 7f3de554f640 1 -- 172.21.15.112:0/746906765 >> v1:172.21.15.112:6790/0 conn(0x7f3de00723b0 legacy=0x7f3de0146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:24.597 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.616+0000 7f3de554f640 1 -- 172.21.15.112:0/746906765 >> v1:172.21.15.118:6789/0 conn(0x7f3de0146b80 legacy=0x7f3de0151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:54:24.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.616+0000 7f3de554f640 10 monclient: _finish_hunting 0 2024-03-29T10:54:24.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.616+0000 7f3de554f640 1 monclient: found mon.noname-a 2024-03-29T10:54:24.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.616+0000 7f3de554f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:24.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.616+0000 7f3de554f640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T10:54:24.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.616+0000 7f3de554f640 1 -- 172.21.15.112:0/746906765 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de006c7e0 con 0x7f3de006d260 2024-03-29T10:54:24.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.616+0000 7f3de554f640 10 monclient: _finish_auth 0 2024-03-29T10:54:24.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.616+0000 7f3de554f640 10 monclient: _check_auth_tickets 2024-03-29T10:54:24.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.616+0000 7f3de554f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:24.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.616+0000 7f3de554f640 1 -- 172.21.15.112:0/746906765 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f3dd80019f0 con 0x7f3de006d260 2024-03-29T10:54:24.600 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.616+0000 7f3de554f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.600 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.616+0000 7f3de554f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:24.600 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.616+0000 7f3de554f640 10 monclient: dump: 2024-03-29T10:54:24.600 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:24.600 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:24.600 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:24.600 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:24.600 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.600 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:24.600 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:24.600 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:24.600 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:24.600 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:24.600 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.616+0000 7f3de87b5640 5 monclient: authenticate success, global_id 4332 2024-03-29T10:54:24.600 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.616+0000 7f3de87b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:24.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.617+0000 7f3de554f640 1 -- 172.21.15.112:0/746906765 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3dd8001c50 con 0x7f3de006d260 2024-03-29T10:54:24.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.617+0000 7f3de554f640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:24.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.617+0000 7f3de554f640 1 -- 172.21.15.112:0/746906765 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f3dd80020a0 con 0x7f3de006d260 2024-03-29T10:54:24.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.617+0000 7f3de554f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.617+0000 7f3de554f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:24.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.617+0000 7f3de554f640 10 monclient: dump: 2024-03-29T10:54:24.601 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:24.601 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:24.601 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:24.601 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:24.601 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.601 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:24.601 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:24.601 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:24.601 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:24.602 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:24.602 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.617+0000 7f3de87b5640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:24.602 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.617+0000 7f3de87b5640 4 set_mon_vals no callback set 2024-03-29T10:54:24.602 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.618+0000 7f3de87b5640 10 monclient: shutdown 2024-03-29T10:54:24.602 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.618+0000 7f3de87b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:24.602 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.618+0000 7f3de87b5640 1 -- 172.21.15.112:0/746906765 >> v1:172.21.15.112:6789/0 conn(0x7f3de006d260 legacy=0x7f3de0071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.619+0000 7f3de87b5640 1 -- 172.21.15.112:0/746906765 shutdown_connections 2024-03-29T10:54:24.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.619+0000 7f3de87b5640 1 -- 172.21.15.112:0/746906765 >> 172.21.15.112:0/746906765 conn(0x7f3de0067500 msgr2=0x7f3de00b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:24.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.619+0000 7f3de87b5640 1 -- 172.21.15.112:0/746906765 shutdown_connections 2024-03-29T10:54:24.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.620+0000 7f3de87b5640 1 -- 172.21.15.112:0/746906765 wait complete. 2024-03-29T10:54:24.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.621+0000 7f3de87b5640 10 monclient: build_initial_monmap 2024-03-29T10:54:24.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.621+0000 7f3de87b5640 10 monclient: monmap: 2024-03-29T10:54:24.603 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:54:24.603 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:24.603 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:54:24.603 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:54:24.603 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:24.604 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:24.604 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:54:24.604 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:54:24.604 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:54:24.604 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:24.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.622+0000 7f3de87b5640 1 librados: starting msgr at 2024-03-29T10:54:24.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.623+0000 7f3de87b5640 1 librados: starting objecter 2024-03-29T10:54:24.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.623+0000 7f3de87b5640 1 Processor -- start 2024-03-29T10:54:24.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.624+0000 7f3de87b5640 1 -- start start 2024-03-29T10:54:24.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.624+0000 7f3de87b5640 1 librados: setting wanted keys 2024-03-29T10:54:24.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.624+0000 7f3de87b5640 1 librados: calling monclient init 2024-03-29T10:54:24.605 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.624+0000 7f3de87b5640 10 monclient: init 2024-03-29T10:54:24.606 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.625+0000 7f3de87b5640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:24.606 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.625+0000 7f3de87b5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:24.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.625+0000 7f3de87b5640 10 monclient(hunting): picked mon.noname-c con 0x7f3de006d260 addr v1:172.21.15.112:6790/0 2024-03-29T10:54:24.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.625+0000 7f3de87b5640 10 monclient(hunting): picked mon.noname-a con 0x7f3de00723b0 addr v1:172.21.15.112:6789/0 2024-03-29T10:54:24.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.625+0000 7f3de87b5640 10 monclient(hunting): picked mon.noname-b con 0x7f3de0146b80 addr v1:172.21.15.118:6789/0 2024-03-29T10:54:24.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.625+0000 7f3de87b5640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de0148aa0 con 0x7f3de00723b0 2024-03-29T10:54:24.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.625+0000 7f3de87b5640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de00ac1a0 con 0x7f3de0146b80 2024-03-29T10:54:24.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.625+0000 7f3de87b5640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de00a9e20 con 0x7f3de006d260 2024-03-29T10:54:24.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.625+0000 7f3de87b5640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:24.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.625+0000 7f3de87b5640 10 monclient(hunting): authenticate will time out at 1019.930020s 2024-03-29T10:54:24.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.626+0000 7f3de5d50640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f3de00723b0 0x7f3de0149640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36976/0 (socket says 172.21.15.112:36976) 2024-03-29T10:54:24.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.626+0000 7f3de5d50640 1 -- 172.21.15.112:0/657697629 learned_addr learned my addr 172.21.15.112:0/657697629 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:54:24.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.627+0000 7f3dd6ffd640 10 client.?.objecter ms_handle_connect 0x7f3de00723b0 2024-03-29T10:54:24.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.627+0000 7f3dd6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:24.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.627+0000 7f3dd6ffd640 1 -- 172.21.15.112:0/657697629 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2296953594 0 0) 0x7f3de0148aa0 con 0x7f3de00723b0 2024-03-29T10:54:24.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.627+0000 7f3dd6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.627+0000 7f3dd6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.627+0000 7f3dd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.627+0000 7f3dd6ffd640 10 monclient(hunting): my global_id is 4335 2024-03-29T10:54:24.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.627+0000 7f3dd6ffd640 1 -- 172.21.15.112:0/657697629 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dc4004eb0 con 0x7f3de00723b0 2024-03-29T10:54:24.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.627+0000 7f3dd6ffd640 1 -- 172.21.15.112:0/657697629 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 556593471 0 0) 0x7f3dc4004eb0 con 0x7f3de00723b0 2024-03-29T10:54:24.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.627+0000 7f3dd6ffd640 1 -- 172.21.15.112:0/657697629 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3de0148aa0 con 0x7f3de00723b0 2024-03-29T10:54:24.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.627+0000 7f3dd6ffd640 1 -- 172.21.15.112:0/657697629 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f3ddc001930 con 0x7f3de00723b0 2024-03-29T10:54:24.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.627+0000 7f3dd6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.627+0000 7f3dd6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:24.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.627+0000 7f3dd6ffd640 10 monclient(hunting): dump: 2024-03-29T10:54:24.610 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:24.610 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:24.610 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:24.610 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:24.610 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.610 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:24.610 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:24.611 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:24.611 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:24.611 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:24.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.627+0000 7f3dd6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:24.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.630+0000 7f3dd6ffd640 10 client.?.objecter ms_handle_connect 0x7f3de0146b80 2024-03-29T10:54:24.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.630+0000 7f3dd6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:24.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.630+0000 7f3dd6ffd640 1 -- 172.21.15.112:0/657697629 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 757712752 0 0) 0x7f3de0148aa0 con 0x7f3de00723b0 2024-03-29T10:54:24.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.630+0000 7f3dd6ffd640 1 -- 172.21.15.112:0/657697629 >> v1:172.21.15.112:6790/0 conn(0x7f3de006d260 legacy=0x7f3de0148f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:54:24.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.632+0000 7f3dd6ffd640 1 -- 172.21.15.112:0/657697629 >> v1:172.21.15.118:6789/0 conn(0x7f3de0146b80 legacy=0x7f3de01e6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.614 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.632+0000 7f3dd6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:54:24.614 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.632+0000 7f3dd6ffd640 1 monclient: found mon.a 2024-03-29T10:54:24.614 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.632+0000 7f3dd6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:24.614 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.632+0000 7f3dd6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:54:24.614 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.632+0000 7f3dd6ffd640 1 -- 172.21.15.112:0/657697629 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de01df9a0 con 0x7f3de00723b0 2024-03-29T10:54:24.614 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.632+0000 7f3dd6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:54:24.614 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.632+0000 7f3dd6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:54:24.614 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.632+0000 7f3dd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:24.614 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.634+0000 7f3dd6ffd640 1 -- 172.21.15.112:0/657697629 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ddc001a90 con 0x7f3de00723b0 2024-03-29T10:54:24.615 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.634+0000 7f3dd6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:24.615 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.634+0000 7f3dd6ffd640 1 -- 172.21.15.112:0/657697629 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f3ddc003a40 con 0x7f3de00723b0 2024-03-29T10:54:24.615 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.634+0000 7f3dd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.615 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.634+0000 7f3dd6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:24.615 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.634+0000 7f3dd6ffd640 10 monclient: dump: 2024-03-29T10:54:24.615 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:24.615 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:24.615 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:24.615 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:24.615 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.615 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:24.615 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:24.615 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:24.615 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:24.616 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:24.616 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.634+0000 7f3de4d4e640 4 set_mon_vals no callback set 2024-03-29T10:54:24.616 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.634+0000 7f3de87b5640 5 monclient: authenticate success, global_id 4335 2024-03-29T10:54:24.617 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.634+0000 7f3de87b5640 10 monclient: _renew_subs 2024-03-29T10:54:24.617 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.634+0000 7f3de87b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:54:24.617 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.634+0000 7f3de87b5640 1 -- 172.21.15.112:0/657697629 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de00ab210 con 0x7f3de00723b0 2024-03-29T10:54:24.617 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.635+0000 7f3de87b5640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:24.617 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.635+0000 7f3de87b5640 10 monclient: _renew_subs 2024-03-29T10:54:24.617 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.635+0000 7f3de87b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:54:24.617 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.635+0000 7f3de87b5640 1 -- 172.21.15.112:0/657697629 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de01df160 con 0x7f3de00723b0 2024-03-29T10:54:24.617 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.635+0000 7f3de87b5640 1 librados: init done 2024-03-29T10:54:24.617 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.636+0000 7f3de87b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:24.618 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.636+0000 7f3de87b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:24.618 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.636+0000 7f3de87b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:54:24.618 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.636+0000 7f3de87b5640 1 -- 172.21.15.112:0/657697629 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de01dfb90 con 0x7f3de00723b0 2024-03-29T10:54:24.621 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.639+0000 7f3dd6ffd640 1 -- 172.21.15.112:0/657697629 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f3ddc003ba0 con 0x7f3de00723b0 2024-03-29T10:54:24.621 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.640+0000 7f3dd6ffd640 1 -- 172.21.15.112:0/657697629 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f3ddc084b00 con 0x7f3de00723b0 2024-03-29T10:54:24.621 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.640+0000 7f3dd6ffd640 10 client.4335.objecter ms_dispatch 0x7f3de0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:24.621 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.640+0000 7f3dd6ffd640 3 client.4335.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:24.621 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.640+0000 7f3dd6ffd640 3 client.4335.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:24.622 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.640+0000 7f3dd6ffd640 20 client.4335.objecter dump_active .. 0 homeless 2024-03-29T10:54:24.624 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.643+0000 7f3de6551640 10 monclient: get_auth_request con 0x7f3dc406dc10 auth_method 0 2024-03-29T10:54:24.625 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.644+0000 7f3dd6ffd640 10 client.4335.objecter ms_handle_connect 0x7f3dc406dc10 2024-03-29T10:54:24.629 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.648+0000 7f3dd6ffd640 1 -- 172.21.15.112:0/657697629 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3ddc0503e0 con 0x7f3de00723b0 2024-03-29T10:54:24.629 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.648+0000 7f3dd6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:24.629 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.648+0000 7f3dd6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:24.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.967+0000 7f3de87b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:24.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.967+0000 7f3de87b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:24.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.967+0000 7f3de87b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:54:24.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.967+0000 7f3de87b5640 1 -- 172.21.15.112:0/657697629 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3de0148c50 con 0x7f3de00723b0 2024-03-29T10:54:24.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.968+0000 7f3dd6ffd640 1 -- 172.21.15.112:0/657697629 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f3de0148c50 con 0x7f3de00723b0 2024-03-29T10:54:24.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.968+0000 7f3dd6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:24.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.968+0000 7f3dd6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:24.950 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:54:24.950 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:24.953 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.971+0000 7f3de87b5640 10 librados: watch_flush enter 2024-03-29T10:54:24.953 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.971+0000 7f3de87b5640 10 librados: watch_flush exit 2024-03-29T10:54:24.953 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.971+0000 7f3de87b5640 20 client.4335.objecter shutdown clearing up homeless session... 2024-03-29T10:54:24.953 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.971+0000 7f3de87b5640 10 client.4335.objecter successfully canceled tick 2024-03-29T10:54:24.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.973+0000 7f3de87b5640 1 -- 172.21.15.112:0/657697629 >> v1:172.21.15.118:6816/3181073408 conn(0x7f3dc406dc10 legacy=0x7f3dc40700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.973+0000 7f3de87b5640 10 monclient: shutdown 2024-03-29T10:54:24.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.973+0000 7f3de87b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:24.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.973+0000 7f3de87b5640 1 -- 172.21.15.112:0/657697629 >> v1:172.21.15.112:6789/0 conn(0x7f3de00723b0 legacy=0x7f3de0149640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.955 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.974+0000 7f3de87b5640 1 -- 172.21.15.112:0/657697629 shutdown_connections 2024-03-29T10:54:24.955 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.974+0000 7f3de87b5640 1 -- 172.21.15.112:0/657697629 >> 172.21.15.112:0/657697629 conn(0x7f3de0067500 msgr2=0x7f3de00707b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:24.956 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.975+0000 7f3de87b5640 1 -- 172.21.15.112:0/657697629 shutdown_connections 2024-03-29T10:54:24.958 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.976+0000 7f3de87b5640 1 -- 172.21.15.112:0/657697629 wait complete. 2024-03-29T10:54:24.958 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:24.976+0000 7f3de87b5640 1 librados: shutdown 2024-03-29T10:54:24.975 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:31.782 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:31.940 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.959+0000 7fa896b6c640 10 monclient: get_monmap_and_config 2024-03-29T10:54:31.940 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.959+0000 7fa896b6c640 10 monclient: build_initial_monmap 2024-03-29T10:54:31.940 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.959+0000 7fa896b6c640 10 monclient: monmap: 2024-03-29T10:54:31.940 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:54:31.940 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:31.941 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:54:31.960100+0000 2024-03-29T10:54:31.941 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:54:31.960100+0000 2024-03-29T10:54:31.941 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:31.941 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:31.941 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:54:31.941 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:54:31.941 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:54:31.941 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:31.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.960+0000 7fa896b6c640 1 Processor -- start 2024-03-29T10:54:31.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.960+0000 7fa896b6c640 1 -- start start 2024-03-29T10:54:31.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.960+0000 7fa896b6c640 10 monclient: init 2024-03-29T10:54:31.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.961+0000 7fa896b6c640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:31.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.961+0000 7fa896b6c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:54:31.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.962+0000 7fa896b6c640 10 monclient(hunting): picked mon.noname-a con 0x7fa89006ade0 addr v1:172.21.15.112:6789/0 2024-03-29T10:54:31.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.962+0000 7fa894908640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fa89006ade0 0x7fa8901499e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33166/0 (socket says 172.21.15.112:33166) 2024-03-29T10:54:31.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.962+0000 7fa894908640 1 -- 172.21.15.112:0/1526899967 learned_addr learned my addr 172.21.15.112:0/1526899967 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:54:31.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.962+0000 7fa896b6c640 10 monclient(hunting): picked mon.noname-b con 0x7fa89006ed60 addr v1:172.21.15.112:6790/0 2024-03-29T10:54:31.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.962+0000 7fa896b6c640 10 monclient(hunting): picked mon.noname-c con 0x7fa890072340 addr v1:172.21.15.118:6789/0 2024-03-29T10:54:31.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.962+0000 7fa896b6c640 1 -- 172.21.15.112:0/1526899967 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa89006ff50 con 0x7fa89006ade0 2024-03-29T10:54:31.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.962+0000 7fa896b6c640 1 -- 172.21.15.112:0/1526899967 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa890070180 con 0x7fa890072340 2024-03-29T10:54:31.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.962+0000 7fa896b6c640 1 -- 172.21.15.112:0/1526899967 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa89006f310 con 0x7fa89006ed60 2024-03-29T10:54:31.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.962+0000 7fa896b6c640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:31.951 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.962+0000 7fa896b6c640 10 monclient(hunting): authenticate will time out at 1027.267168s 2024-03-29T10:54:31.951 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.963+0000 7fa88f7fe640 1 -- 172.21.15.112:0/1526899967 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3834751886 0 0) 0x7fa890070180 con 0x7fa890072340 2024-03-29T10:54:31.951 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.963+0000 7fa88f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:31.951 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.963+0000 7fa88f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:31.951 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.963+0000 7fa88f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:31.951 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.963+0000 7fa88f7fe640 10 monclient(hunting): my global_id is 4333 2024-03-29T10:54:31.951 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.963+0000 7fa88f7fe640 1 -- 172.21.15.112:0/1526899967 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa87c003520 con 0x7fa890072340 2024-03-29T10:54:31.951 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 1 -- 172.21.15.112:0/1526899967 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373364764 0 0) 0x7fa89006ff50 con 0x7fa89006ade0 2024-03-29T10:54:31.951 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:31.951 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:31.951 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:31.952 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 10 monclient(hunting): my global_id is 4338 2024-03-29T10:54:31.952 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 1 -- 172.21.15.112:0/1526899967 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa890070180 con 0x7fa89006ade0 2024-03-29T10:54:31.952 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 1 -- 172.21.15.112:0/1526899967 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3822908625 0 0) 0x7fa87c003520 con 0x7fa890072340 2024-03-29T10:54:31.952 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 1 -- 172.21.15.112:0/1526899967 >> v1:172.21.15.112:6790/0 conn(0x7fa89006ed60 legacy=0x7fa890071c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:31.952 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 1 -- 172.21.15.112:0/1526899967 >> v1:172.21.15.112:6789/0 conn(0x7fa89006ade0 legacy=0x7fa8901499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:31.952 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:54:31.952 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 1 monclient: found mon.noname-c 2024-03-29T10:54:31.952 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:31.952 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T10:54:31.952 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 1 -- 172.21.15.112:0/1526899967 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8900aeed0 con 0x7fa890072340 2024-03-29T10:54:31.952 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:54:31.952 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:54:31.953 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:31.953 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 1 -- 172.21.15.112:0/1526899967 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fa888001690 con 0x7fa890072340 2024-03-29T10:54:31.953 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:31.953 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:31.953 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 10 monclient: dump: 2024-03-29T10:54:31.953 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:31.953 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:31.953 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:31.953 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:31.953 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:31.953 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:31.953 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:31.953 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:31.954 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:31.954 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:31.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa896b6c640 5 monclient: authenticate success, global_id 4333 2024-03-29T10:54:31.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa896b6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:31.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 1 -- 172.21.15.112:0/1526899967 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8880018b0 con 0x7fa890072340 2024-03-29T10:54:31.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:31.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 1 -- 172.21.15.112:0/1526899967 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fa888001c40 con 0x7fa890072340 2024-03-29T10:54:31.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.964+0000 7fa88f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:31.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.965+0000 7fa88f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:31.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.965+0000 7fa88f7fe640 10 monclient: dump: 2024-03-29T10:54:31.954 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:31.954 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:31.954 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:31.955 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:31.955 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:31.955 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:31.955 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:31.955 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:31.955 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:31.955 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:31.955 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.965+0000 7fa896b6c640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:31.955 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.965+0000 7fa896b6c640 4 set_mon_vals no callback set 2024-03-29T10:54:31.955 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.965+0000 7fa896b6c640 10 monclient: shutdown 2024-03-29T10:54:31.955 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.966+0000 7fa896b6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:31.955 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.966+0000 7fa896b6c640 1 -- 172.21.15.112:0/1526899967 >> v1:172.21.15.118:6789/0 conn(0x7fa890072340 legacy=0x7fa890151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:31.955 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.966+0000 7fa896b6c640 1 -- 172.21.15.112:0/1526899967 shutdown_connections 2024-03-29T10:54:31.955 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.966+0000 7fa896b6c640 1 -- 172.21.15.112:0/1526899967 >> 172.21.15.112:0/1526899967 conn(0x7fa8900b4930 msgr2=0x7fa8900b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:31.955 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.966+0000 7fa896b6c640 1 -- 172.21.15.112:0/1526899967 shutdown_connections 2024-03-29T10:54:31.955 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.966+0000 7fa896b6c640 1 -- 172.21.15.112:0/1526899967 wait complete. 2024-03-29T10:54:31.956 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.967+0000 7fa896b6c640 10 monclient: build_initial_monmap 2024-03-29T10:54:31.956 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.967+0000 7fa896b6c640 10 monclient: monmap: 2024-03-29T10:54:31.956 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:54:31.956 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:31.956 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:54:31.956 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:54:31.956 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:31.956 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:31.956 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:54:31.956 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:54:31.956 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:54:31.956 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:31.956 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.967+0000 7fa896b6c640 1 librados: starting msgr at 2024-03-29T10:54:31.956 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.967+0000 7fa896b6c640 1 librados: starting objecter 2024-03-29T10:54:31.957 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.967+0000 7fa896b6c640 1 Processor -- start 2024-03-29T10:54:31.957 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.968+0000 7fa896b6c640 1 -- start start 2024-03-29T10:54:31.957 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.968+0000 7fa896b6c640 1 librados: setting wanted keys 2024-03-29T10:54:31.957 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.968+0000 7fa896b6c640 1 librados: calling monclient init 2024-03-29T10:54:31.957 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.968+0000 7fa896b6c640 10 monclient: init 2024-03-29T10:54:31.960 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.969+0000 7fa896b6c640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:31.961 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.969+0000 7fa896b6c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:54:31.961 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.969+0000 7fa896b6c640 10 monclient(hunting): picked mon.noname-b con 0x7fa89006ade0 addr v1:172.21.15.118:6789/0 2024-03-29T10:54:31.961 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.969+0000 7fa896b6c640 10 monclient(hunting): picked mon.noname-a con 0x7fa89006ed60 addr v1:172.21.15.112:6789/0 2024-03-29T10:54:31.961 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.969+0000 7fa896b6c640 10 monclient(hunting): picked mon.noname-c con 0x7fa890072340 addr v1:172.21.15.112:6790/0 2024-03-29T10:54:31.961 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.969+0000 7fa896b6c640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8901db390 con 0x7fa89006ed60 2024-03-29T10:54:31.961 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.969+0000 7fa896b6c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8900aad10 con 0x7fa89006ade0 2024-03-29T10:54:31.961 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.969+0000 7fa896b6c640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa890145070 con 0x7fa890072340 2024-03-29T10:54:31.961 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.969+0000 7fa896b6c640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:31.961 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.969+0000 7fa896b6c640 10 monclient(hunting): authenticate will time out at 1027.273490s 2024-03-29T10:54:31.961 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.969+0000 7fa894908640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fa89006ade0 0x7fa8900aa600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:40272/0 (socket says 172.21.15.112:40272) 2024-03-29T10:54:31.961 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.969+0000 7fa894908640 1 -- 172.21.15.112:0/28797666 learned_addr learned my addr 172.21.15.112:0/28797666 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:54:31.961 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.969+0000 7fa895109640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fa890072340 0x7fa8901e5df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:34094/0 (socket says 172.21.15.112:34094) 2024-03-29T10:54:31.961 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.969+0000 7fa88d7fa640 10 client.?.objecter ms_handle_connect 0x7fa89006ade0 2024-03-29T10:54:31.961 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.969+0000 7fa88d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:31.962 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.969+0000 7fa88d7fa640 10 client.?.objecter ms_handle_connect 0x7fa890072340 2024-03-29T10:54:31.962 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.969+0000 7fa88d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:31.962 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.970+0000 7fa88d7fa640 10 client.?.objecter ms_handle_connect 0x7fa89006ed60 2024-03-29T10:54:31.962 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.970+0000 7fa88d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:31.962 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.970+0000 7fa88d7fa640 1 -- 172.21.15.112:0/28797666 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2110386442 0 0) 0x7fa8900aad10 con 0x7fa89006ade0 2024-03-29T10:54:31.962 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.970+0000 7fa88d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:31.962 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.970+0000 7fa88d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:31.962 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.970+0000 7fa88d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:31.962 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.970+0000 7fa88d7fa640 10 monclient(hunting): my global_id is 4336 2024-03-29T10:54:31.962 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.970+0000 7fa88d7fa640 1 -- 172.21.15.112:0/28797666 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa870004eb0 con 0x7fa89006ade0 2024-03-29T10:54:31.963 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.970+0000 7fa88d7fa640 1 -- 172.21.15.112:0/28797666 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385100819 0 0) 0x7fa890145070 con 0x7fa890072340 2024-03-29T10:54:31.963 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.970+0000 7fa88d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:31.963 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.970+0000 7fa88d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:31.963 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.970+0000 7fa88d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:31.963 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.970+0000 7fa88d7fa640 10 monclient(hunting): my global_id is 4343 2024-03-29T10:54:31.963 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.970+0000 7fa88d7fa640 1 -- 172.21.15.112:0/28797666 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8900aad10 con 0x7fa890072340 2024-03-29T10:54:31.963 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.971+0000 7fa88d7fa640 1 -- 172.21.15.112:0/28797666 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2263605815 0 0) 0x7fa870004eb0 con 0x7fa89006ade0 2024-03-29T10:54:31.963 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.971+0000 7fa88d7fa640 1 -- 172.21.15.112:0/28797666 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa890145070 con 0x7fa89006ade0 2024-03-29T10:54:31.963 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.971+0000 7fa88d7fa640 1 -- 172.21.15.112:0/28797666 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fa880001d30 con 0x7fa89006ade0 2024-03-29T10:54:31.963 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.971+0000 7fa88d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:31.963 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.971+0000 7fa88d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:31.963 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.971+0000 7fa88d7fa640 10 monclient(hunting): dump: 2024-03-29T10:54:31.963 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:31.963 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:31.964 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:31.964 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:31.964 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:31.964 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:31.964 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:31.964 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:31.964 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:31.964 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:31.964 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.971+0000 7fa88d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:31.964 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.971+0000 7fa88d7fa640 1 -- 172.21.15.112:0/28797666 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632202867 0 0) 0x7fa8901db390 con 0x7fa89006ed60 2024-03-29T10:54:31.964 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.971+0000 7fa88d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:31.964 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.971+0000 7fa88d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:31.964 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.971+0000 7fa88d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:31.964 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.971+0000 7fa88d7fa640 10 monclient(hunting): my global_id is 4341 2024-03-29T10:54:31.965 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.971+0000 7fa88d7fa640 1 -- 172.21.15.112:0/28797666 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa870004eb0 con 0x7fa89006ed60 2024-03-29T10:54:31.965 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.971+0000 7fa88d7fa640 1 -- 172.21.15.112:0/28797666 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3502950413 0 0) 0x7fa8900aad10 con 0x7fa890072340 2024-03-29T10:54:31.965 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa88d7fa640 1 -- 172.21.15.112:0/28797666 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8901db390 con 0x7fa890072340 2024-03-29T10:54:31.965 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa88d7fa640 1 -- 172.21.15.112:0/28797666 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fa888001bc0 con 0x7fa890072340 2024-03-29T10:54:31.965 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa88d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:31.965 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa88d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:31.965 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa88d7fa640 10 monclient(hunting): dump: 2024-03-29T10:54:31.965 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:31.965 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:31.965 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:31.965 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:31.965 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:31.965 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:31.965 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:31.966 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:31.966 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:31.966 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:31.966 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa88d7fa640 1 -- 172.21.15.112:0/28797666 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 54088380 0 0) 0x7fa890145070 con 0x7fa89006ade0 2024-03-29T10:54:31.966 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa88d7fa640 1 -- 172.21.15.112:0/28797666 >> v1:172.21.15.112:6790/0 conn(0x7fa890072340 legacy=0x7fa8901e5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:31.966 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa88d7fa640 1 -- 172.21.15.112:0/28797666 >> v1:172.21.15.112:6789/0 conn(0x7fa89006ed60 legacy=0x7fa8901db8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:31.966 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa88d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:54:31.966 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa88d7fa640 1 monclient: found mon.b 2024-03-29T10:54:31.966 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa88d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:31.966 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa88d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:54:31.966 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa88d7fa640 1 -- 172.21.15.112:0/28797666 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa890145200 con 0x7fa89006ade0 2024-03-29T10:54:31.966 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa88d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:54:31.966 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa88d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:54:31.966 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa88d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:31.966 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa896b6c640 5 monclient: authenticate success, global_id 4336 2024-03-29T10:54:31.967 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa896b6c640 10 monclient: _renew_subs 2024-03-29T10:54:31.967 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa896b6c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:54:31.967 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa896b6c640 1 -- 172.21.15.112:0/28797666 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8901dbfc0 con 0x7fa89006ade0 2024-03-29T10:54:31.967 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa896b6c640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:31.967 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa896b6c640 10 monclient: _renew_subs 2024-03-29T10:54:31.967 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa896b6c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:54:31.967 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa896b6c640 1 -- 172.21.15.112:0/28797666 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8900a9d70 con 0x7fa89006ade0 2024-03-29T10:54:31.967 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.972+0000 7fa896b6c640 1 librados: init done 2024-03-29T10:54:31.967 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.973+0000 7fa88d7fa640 1 -- 172.21.15.112:0/28797666 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa880004b60 con 0x7fa89006ade0 2024-03-29T10:54:31.967 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.973+0000 7fa88d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:31.967 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.973+0000 7fa88d7fa640 1 -- 172.21.15.112:0/28797666 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fa880004e70 con 0x7fa89006ade0 2024-03-29T10:54:31.967 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.973+0000 7fa88d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:31.967 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.973+0000 7fa88d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:31.967 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.973+0000 7fa88d7fa640 10 monclient: dump: 2024-03-29T10:54:31.968 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:31.968 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:31.968 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:31.968 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:31.968 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:31.968 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:31.968 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:31.968 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:31.968 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:31.968 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:31.968 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.973+0000 7fa88effd640 4 set_mon_vals no callback set 2024-03-29T10:54:31.968 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.975+0000 7fa88d7fa640 1 -- 172.21.15.112:0/28797666 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fa88001ae40 con 0x7fa89006ade0 2024-03-29T10:54:31.969 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.975+0000 7fa88d7fa640 1 -- 172.21.15.112:0/28797666 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7fa880084ea0 con 0x7fa89006ade0 2024-03-29T10:54:31.969 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.975+0000 7fa88d7fa640 10 client.4336.objecter ms_dispatch 0x7fa890000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:31.969 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.975+0000 7fa88d7fa640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:31.969 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.975+0000 7fa88d7fa640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:31.969 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.975+0000 7fa88d7fa640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-29T10:54:31.969 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.976+0000 7fa88ffff640 10 monclient: get_auth_request con 0x7fa87006df20 auth_method 0 2024-03-29T10:54:31.969 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.976+0000 7fa88d7fa640 10 client.4336.objecter ms_handle_connect 0x7fa87006df20 2024-03-29T10:54:31.969 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.977+0000 7fa896b6c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:31.969 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.977+0000 7fa896b6c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:31.969 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.977+0000 7fa896b6c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:54:31.969 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.977+0000 7fa896b6c640 1 -- 172.21.15.112:0/28797666 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa874005080 con 0x7fa89006ade0 2024-03-29T10:54:31.970 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.984+0000 7fa88d7fa640 1 -- 172.21.15.112:0/28797666 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa8800505b0 con 0x7fa89006ade0 2024-03-29T10:54:31.970 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.984+0000 7fa88d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:31.970 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:31.984+0000 7fa88d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:32.354 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:32.373+0000 7fa896b6c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:32.354 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:32.373+0000 7fa896b6c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:32.355 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:32.373+0000 7fa896b6c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:54:32.355 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:32.373+0000 7fa896b6c640 1 -- 172.21.15.112:0/28797666 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa874005330 con 0x7fa89006ade0 2024-03-29T10:54:32.355 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:32.374+0000 7fa88d7fa640 1 -- 172.21.15.112:0/28797666 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7fa8800526b0 con 0x7fa89006ade0 2024-03-29T10:54:32.355 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:32.374+0000 7fa88d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:32.355 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:32.374+0000 7fa88d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:32.356 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:54:32.356 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:32.359 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:32.378+0000 7fa896b6c640 10 librados: watch_flush enter 2024-03-29T10:54:32.359 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:32.378+0000 7fa896b6c640 10 librados: watch_flush exit 2024-03-29T10:54:32.359 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:32.378+0000 7fa896b6c640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-29T10:54:32.359 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:32.378+0000 7fa896b6c640 10 client.4336.objecter successfully canceled tick 2024-03-29T10:54:32.359 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:32.378+0000 7fa896b6c640 1 -- 172.21.15.112:0/28797666 >> v1:172.21.15.118:6816/3181073408 conn(0x7fa87006df20 legacy=0x7fa8700703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:32.359 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:32.378+0000 7fa896b6c640 10 monclient: shutdown 2024-03-29T10:54:32.359 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:32.378+0000 7fa896b6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:32.359 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:32.378+0000 7fa896b6c640 1 -- 172.21.15.112:0/28797666 >> v1:172.21.15.118:6789/0 conn(0x7fa89006ade0 legacy=0x7fa8900aa600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:32.360 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:32.378+0000 7fa896b6c640 1 -- 172.21.15.112:0/28797666 shutdown_connections 2024-03-29T10:54:32.360 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:32.378+0000 7fa896b6c640 1 -- 172.21.15.112:0/28797666 >> 172.21.15.112:0/28797666 conn(0x7fa8900b4930 msgr2=0x7fa8900b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:32.360 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:32.379+0000 7fa896b6c640 1 -- 172.21.15.112:0/28797666 shutdown_connections 2024-03-29T10:54:32.360 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:32.379+0000 7fa896b6c640 1 -- 172.21.15.112:0/28797666 wait complete. 2024-03-29T10:54:32.361 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:32.379+0000 7fa896b6c640 1 librados: shutdown 2024-03-29T10:54:32.375 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:39.182 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:39.333 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.351+0000 7fc3b2d3b640 10 monclient: get_monmap_and_config 2024-03-29T10:54:39.333 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.351+0000 7fc3b2d3b640 10 monclient: build_initial_monmap 2024-03-29T10:54:39.333 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.351+0000 7fc3b2d3b640 10 monclient: monmap: 2024-03-29T10:54:39.334 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:54:39.334 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:39.334 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:54:39.352391+0000 2024-03-29T10:54:39.334 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:54:39.352391+0000 2024-03-29T10:54:39.334 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:39.334 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:39.334 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:54:39.334 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:54:39.334 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:54:39.335 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:39.335 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.352+0000 7fc3b2d3b640 1 Processor -- start 2024-03-29T10:54:39.335 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.352+0000 7fc3b2d3b640 1 -- start start 2024-03-29T10:54:39.335 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.352+0000 7fc3b2d3b640 10 monclient: init 2024-03-29T10:54:39.339 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.354+0000 7fc3b2d3b640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:39.339 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.354+0000 7fc3b2d3b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:54:39.339 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.354+0000 7fc3b2d3b640 10 monclient(hunting): picked mon.noname-b con 0x7fc3ac1457f0 addr v1:172.21.15.112:6790/0 2024-03-29T10:54:39.339 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.354+0000 7fc3b2d3b640 10 monclient(hunting): picked mon.noname-c con 0x7fc3ac13f650 addr v1:172.21.15.118:6789/0 2024-03-29T10:54:39.339 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.354+0000 7fc3b2d3b640 10 monclient(hunting): picked mon.noname-a con 0x7fc3ac0a9d70 addr v1:172.21.15.112:6789/0 2024-03-29T10:54:39.339 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.354+0000 7fc3b2d3b640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3ac0aaef0 con 0x7fc3ac0a9d70 2024-03-29T10:54:39.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.354+0000 7fc3b2d3b640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3ac145bd0 con 0x7fc3ac13f650 2024-03-29T10:54:39.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.354+0000 7fc3b2d3b640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3ac145eb0 con 0x7fc3ac1457f0 2024-03-29T10:54:39.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.354+0000 7fc3b2d3b640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:39.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.354+0000 7fc3b2d3b640 10 monclient(hunting): authenticate will time out at 1034.658306s 2024-03-29T10:54:39.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.354+0000 7fc3abfff640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fc3ac13f650 0x7fc3ac0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:48070/0 (socket says 172.21.15.112:48070) 2024-03-29T10:54:39.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.354+0000 7fc3b12d8640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fc3ac0a9d70 0x7fc3ac14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:37906/0 (socket says 172.21.15.112:37906) 2024-03-29T10:54:39.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.354+0000 7fc3b12d8640 1 -- 172.21.15.112:0/3865717889 learned_addr learned my addr 172.21.15.112:0/3865717889 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:54:39.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.354+0000 7fc3ab7fe640 1 -- 172.21.15.112:0/3865717889 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4108130149 0 0) 0x7fc3ac145bd0 con 0x7fc3ac13f650 2024-03-29T10:54:39.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.354+0000 7fc3ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:39.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.354+0000 7fc3ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:39.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.354+0000 7fc3ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:39.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 10 monclient(hunting): my global_id is 4339 2024-03-29T10:54:39.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 1 -- 172.21.15.112:0/3865717889 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc39c003520 con 0x7fc3ac13f650 2024-03-29T10:54:39.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 1 -- 172.21.15.112:0/3865717889 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052091918 0 0) 0x7fc3ac145eb0 con 0x7fc3ac1457f0 2024-03-29T10:54:39.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:39.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:39.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:39.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 10 monclient(hunting): my global_id is 4346 2024-03-29T10:54:39.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 1 -- 172.21.15.112:0/3865717889 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3ac145bd0 con 0x7fc3ac1457f0 2024-03-29T10:54:39.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 1 -- 172.21.15.112:0/3865717889 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1066034785 0 0) 0x7fc3ac145bd0 con 0x7fc3ac1457f0 2024-03-29T10:54:39.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 1 -- 172.21.15.112:0/3865717889 >> v1:172.21.15.118:6789/0 conn(0x7fc3ac13f650 legacy=0x7fc3ac0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:39.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 1 -- 172.21.15.112:0/3865717889 >> v1:172.21.15.112:6789/0 conn(0x7fc3ac0a9d70 legacy=0x7fc3ac14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:39.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:54:39.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 1 monclient: found mon.noname-b 2024-03-29T10:54:39.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:39.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:54:39.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 1 -- 172.21.15.112:0/3865717889 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ac146340 con 0x7fc3ac1457f0 2024-03-29T10:54:39.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 10 monclient: _finish_auth 0 2024-03-29T10:54:39.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:54:39.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:39.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 1 -- 172.21.15.112:0/3865717889 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fc398001670 con 0x7fc3ac1457f0 2024-03-29T10:54:39.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:39.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:39.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 10 monclient: dump: 2024-03-29T10:54:39.342 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:39.343 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:39.343 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:39.343 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:39.343 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:39.343 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:39.343 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:39.343 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:39.343 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:39.343 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:39.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3b2d3b640 5 monclient: authenticate success, global_id 4346 2024-03-29T10:54:39.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3b2d3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:39.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 1 -- 172.21.15.112:0/3865717889 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3980017d0 con 0x7fc3ac1457f0 2024-03-29T10:54:39.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.355+0000 7fc3ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:39.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.356+0000 7fc3b2d3b640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:39.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.356+0000 7fc3b2d3b640 4 set_mon_vals no callback set 2024-03-29T10:54:39.344 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.356+0000 7fc3ab7fe640 1 -- 172.21.15.112:0/3865717889 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fc398001ba0 con 0x7fc3ac1457f0 2024-03-29T10:54:39.344 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.356+0000 7fc3ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:39.344 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.356+0000 7fc3ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:39.344 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.356+0000 7fc3ab7fe640 10 monclient: dump: 2024-03-29T10:54:39.344 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:39.344 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:39.344 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:39.344 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:39.344 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:39.344 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:39.344 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:39.344 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:39.344 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:39.344 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:39.345 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.356+0000 7fc3b2d3b640 10 monclient: shutdown 2024-03-29T10:54:39.345 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.356+0000 7fc3b2d3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:39.345 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.356+0000 7fc3b2d3b640 1 -- 172.21.15.112:0/3865717889 >> v1:172.21.15.112:6790/0 conn(0x7fc3ac1457f0 legacy=0x7fc3ac13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:39.345 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.356+0000 7fc3b2d3b640 1 -- 172.21.15.112:0/3865717889 shutdown_connections 2024-03-29T10:54:39.345 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.356+0000 7fc3b2d3b640 1 -- 172.21.15.112:0/3865717889 >> 172.21.15.112:0/3865717889 conn(0x7fc3ac066a00 msgr2=0x7fc3ac066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:39.345 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.357+0000 7fc3b2d3b640 1 -- 172.21.15.112:0/3865717889 shutdown_connections 2024-03-29T10:54:39.345 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.357+0000 7fc3b2d3b640 1 -- 172.21.15.112:0/3865717889 wait complete. 2024-03-29T10:54:39.345 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.357+0000 7fc3b2d3b640 10 monclient: build_initial_monmap 2024-03-29T10:54:39.345 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.357+0000 7fc3b2d3b640 10 monclient: monmap: 2024-03-29T10:54:39.345 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:54:39.345 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:39.345 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:54:39.345 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:54:39.345 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:39.345 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:39.346 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:54:39.346 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:54:39.346 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:54:39.346 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:39.346 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.358+0000 7fc3b2d3b640 1 librados: starting msgr at 2024-03-29T10:54:39.346 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.358+0000 7fc3b2d3b640 1 librados: starting objecter 2024-03-29T10:54:39.346 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.358+0000 7fc3b2d3b640 1 Processor -- start 2024-03-29T10:54:39.351 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.358+0000 7fc3b2d3b640 1 -- start start 2024-03-29T10:54:39.351 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.358+0000 7fc3b2d3b640 1 librados: setting wanted keys 2024-03-29T10:54:39.351 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.358+0000 7fc3b2d3b640 1 librados: calling monclient init 2024-03-29T10:54:39.351 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.358+0000 7fc3b2d3b640 10 monclient: init 2024-03-29T10:54:39.351 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.359+0000 7fc3b2d3b640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:39.351 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.359+0000 7fc3b2d3b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:54:39.351 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.359+0000 7fc3b2d3b640 10 monclient(hunting): picked mon.noname-b con 0x7fc3ac0a9d70 addr v1:172.21.15.118:6789/0 2024-03-29T10:54:39.352 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.359+0000 7fc3b2d3b640 10 monclient(hunting): picked mon.noname-c con 0x7fc3ac13f650 addr v1:172.21.15.112:6790/0 2024-03-29T10:54:39.352 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.359+0000 7fc3b2d3b640 10 monclient(hunting): picked mon.noname-a con 0x7fc3ac1457f0 addr v1:172.21.15.112:6789/0 2024-03-29T10:54:39.352 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.359+0000 7fc3b2d3b640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3ac13fda0 con 0x7fc3ac1457f0 2024-03-29T10:54:39.352 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.359+0000 7fc3b2d3b640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3ac142d10 con 0x7fc3ac0a9d70 2024-03-29T10:54:39.352 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.359+0000 7fc3abfff640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fc3ac13f650 0x7fc3ac0b0b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41910/0 (socket says 172.21.15.112:41910) 2024-03-29T10:54:39.352 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.359+0000 7fc3abfff640 1 -- 172.21.15.112:0/3744591810 learned_addr learned my addr 172.21.15.112:0/3744591810 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:54:39.352 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.359+0000 7fc3b2d3b640 1 -- 172.21.15.112:0/3744591810 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3ac141f00 con 0x7fc3ac13f650 2024-03-29T10:54:39.352 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.359+0000 7fc3b2d3b640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:39.352 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.359+0000 7fc3b2d3b640 10 monclient(hunting): authenticate will time out at 1034.664168s 2024-03-29T10:54:39.352 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 10 client.?.objecter ms_handle_connect 0x7fc3ac0a9d70 2024-03-29T10:54:39.352 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:39.352 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 10 client.?.objecter ms_handle_connect 0x7fc3ac13f650 2024-03-29T10:54:39.352 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:39.352 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 10 client.?.objecter ms_handle_connect 0x7fc3ac1457f0 2024-03-29T10:54:39.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:39.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 1 -- 172.21.15.112:0/3744591810 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63324779 0 0) 0x7fc3ac13fda0 con 0x7fc3ac1457f0 2024-03-29T10:54:39.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:39.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:39.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:39.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 10 monclient(hunting): my global_id is 4347 2024-03-29T10:54:39.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 1 -- 172.21.15.112:0/3744591810 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc390004eb0 con 0x7fc3ac1457f0 2024-03-29T10:54:39.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 1 -- 172.21.15.112:0/3744591810 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3885602411 0 0) 0x7fc3ac141f00 con 0x7fc3ac13f650 2024-03-29T10:54:39.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:39.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:39.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:39.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 10 monclient(hunting): my global_id is 4349 2024-03-29T10:54:39.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 1 -- 172.21.15.112:0/3744591810 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3ac13fda0 con 0x7fc3ac13f650 2024-03-29T10:54:39.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 1 -- 172.21.15.112:0/3744591810 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1678524024 0 0) 0x7fc3ac142d10 con 0x7fc3ac0a9d70 2024-03-29T10:54:39.354 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:39.354 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:39.354 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:39.354 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 10 monclient(hunting): my global_id is 4342 2024-03-29T10:54:39.354 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 1 -- 172.21.15.112:0/3744591810 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3ac141f00 con 0x7fc3ac0a9d70 2024-03-29T10:54:39.354 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.360+0000 7fc3a97fa640 1 -- 172.21.15.112:0/3744591810 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1628091356 0 0) 0x7fc390004eb0 con 0x7fc3ac1457f0 2024-03-29T10:54:39.354 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3a97fa640 1 -- 172.21.15.112:0/3744591810 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3ac142d10 con 0x7fc3ac1457f0 2024-03-29T10:54:39.354 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3a97fa640 1 -- 172.21.15.112:0/3744591810 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fc3a4001bb0 con 0x7fc3ac1457f0 2024-03-29T10:54:39.354 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:39.354 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3a97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:39.354 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3a97fa640 10 monclient(hunting): dump: 2024-03-29T10:54:39.354 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:39.354 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:39.355 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:39.355 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:39.355 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:39.355 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:39.355 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:39.355 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:39.355 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:39.355 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:39.355 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3a97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:39.355 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3a97fa640 1 -- 172.21.15.112:0/3744591810 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3149953299 0 0) 0x7fc3ac141f00 con 0x7fc3ac0a9d70 2024-03-29T10:54:39.355 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3a97fa640 1 -- 172.21.15.112:0/3744591810 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc390004eb0 con 0x7fc3ac0a9d70 2024-03-29T10:54:39.355 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3a97fa640 1 -- 172.21.15.112:0/3744591810 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fc398001ae0 con 0x7fc3ac0a9d70 2024-03-29T10:54:39.355 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:39.355 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3a97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:39.355 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3a97fa640 10 monclient(hunting): dump: 2024-03-29T10:54:39.355 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:39.356 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:39.356 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:39.356 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:39.356 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:39.356 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:39.356 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:39.356 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:39.356 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:39.356 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:39.356 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3a97fa640 1 -- 172.21.15.112:0/3744591810 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3468923860 0 0) 0x7fc3ac142d10 con 0x7fc3ac1457f0 2024-03-29T10:54:39.356 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3a97fa640 1 -- 172.21.15.112:0/3744591810 >> v1:172.21.15.112:6790/0 conn(0x7fc3ac13f650 legacy=0x7fc3ac0b0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:39.356 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3a97fa640 1 -- 172.21.15.112:0/3744591810 >> v1:172.21.15.118:6789/0 conn(0x7fc3ac0a9d70 legacy=0x7fc3ac142250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:39.356 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3a97fa640 10 monclient: _finish_hunting 0 2024-03-29T10:54:39.356 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3a97fa640 1 monclient: found mon.a 2024-03-29T10:54:39.357 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3a97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:39.357 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3a97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:54:39.357 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3a97fa640 1 -- 172.21.15.112:0/3744591810 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ac142960 con 0x7fc3ac1457f0 2024-03-29T10:54:39.357 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3a97fa640 10 monclient: _finish_auth 0 2024-03-29T10:54:39.357 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3a97fa640 10 monclient: _check_auth_tickets 2024-03-29T10:54:39.357 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:39.357 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3b2d3b640 5 monclient: authenticate success, global_id 4347 2024-03-29T10:54:39.357 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3b2d3b640 10 monclient: _renew_subs 2024-03-29T10:54:39.357 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3b2d3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:54:39.357 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3b2d3b640 1 -- 172.21.15.112:0/3744591810 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3ac13acd0 con 0x7fc3ac1457f0 2024-03-29T10:54:39.357 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3b2d3b640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:39.357 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3b2d3b640 10 monclient: _renew_subs 2024-03-29T10:54:39.357 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3b2d3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:54:39.357 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3b2d3b640 1 -- 172.21.15.112:0/3744591810 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3ac0b1270 con 0x7fc3ac1457f0 2024-03-29T10:54:39.358 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.361+0000 7fc3b2d3b640 1 librados: init done 2024-03-29T10:54:39.358 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.362+0000 7fc3a97fa640 1 -- 172.21.15.112:0/3744591810 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3a4004b60 con 0x7fc3ac1457f0 2024-03-29T10:54:39.358 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.362+0000 7fc3a97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:39.358 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.362+0000 7fc3a97fa640 1 -- 172.21.15.112:0/3744591810 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fc3a4002100 con 0x7fc3ac1457f0 2024-03-29T10:54:39.358 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.362+0000 7fc3a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:39.358 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.362+0000 7fc3a97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:39.358 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.362+0000 7fc3a97fa640 10 monclient: dump: 2024-03-29T10:54:39.358 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:39.358 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:39.358 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:39.358 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:39.358 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:39.358 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:39.358 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:39.359 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:39.359 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:39.359 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:39.359 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.363+0000 7fc3aaffd640 4 set_mon_vals no callback set 2024-03-29T10:54:39.359 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.363+0000 7fc3b2d3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:39.359 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.364+0000 7fc3b2d3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:39.359 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.364+0000 7fc3b2d3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:54:39.359 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.364+0000 7fc3b2d3b640 1 -- 172.21.15.112:0/3744591810 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc38c005080 con 0x7fc3ac1457f0 2024-03-29T10:54:39.359 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.364+0000 7fc3a97fa640 1 -- 172.21.15.112:0/3744591810 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fc3a401ace0 con 0x7fc3ac1457f0 2024-03-29T10:54:39.359 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.365+0000 7fc3a97fa640 1 -- 172.21.15.112:0/3744591810 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7fc3a4084c00 con 0x7fc3ac1457f0 2024-03-29T10:54:39.359 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.365+0000 7fc3a97fa640 10 client.4347.objecter ms_dispatch 0x7fc3ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:39.359 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.365+0000 7fc3a97fa640 3 client.4347.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:39.359 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.365+0000 7fc3a97fa640 3 client.4347.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:39.359 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.365+0000 7fc3a97fa640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-29T10:54:39.359 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.365+0000 7fc3b0ad7640 10 monclient: get_auth_request con 0x7fc39006e050 auth_method 0 2024-03-29T10:54:39.360 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.366+0000 7fc3a97fa640 10 client.4347.objecter ms_handle_connect 0x7fc39006e050 2024-03-29T10:54:39.360 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.371+0000 7fc3a97fa640 1 -- 172.21.15.112:0/3744591810 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc3a40504e0 con 0x7fc3ac1457f0 2024-03-29T10:54:39.360 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.371+0000 7fc3a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:39.360 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.371+0000 7fc3a97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:39.716 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.735+0000 7fc3b2d3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:39.717 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.735+0000 7fc3b2d3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:39.717 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.735+0000 7fc3b2d3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:54:39.717 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.735+0000 7fc3b2d3b640 1 -- 172.21.15.112:0/3744591810 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc38c005330 con 0x7fc3ac1457f0 2024-03-29T10:54:39.717 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.737+0000 7fc3a97fa640 1 -- 172.21.15.112:0/3744591810 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7fc3a4052410 con 0x7fc3ac1457f0 2024-03-29T10:54:39.717 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.737+0000 7fc3a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:39.717 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.737+0000 7fc3a97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:39.718 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:54:39.718 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:39.722 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.740+0000 7fc3b2d3b640 10 librados: watch_flush enter 2024-03-29T10:54:39.722 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.740+0000 7fc3b2d3b640 10 librados: watch_flush exit 2024-03-29T10:54:39.722 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.740+0000 7fc3b2d3b640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-29T10:54:39.722 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.740+0000 7fc3b2d3b640 10 client.4347.objecter successfully canceled tick 2024-03-29T10:54:39.722 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.740+0000 7fc3b2d3b640 1 -- 172.21.15.112:0/3744591810 >> v1:172.21.15.118:6816/3181073408 conn(0x7fc39006e050 legacy=0x7fc390070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:39.722 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.740+0000 7fc3b2d3b640 10 monclient: shutdown 2024-03-29T10:54:39.722 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.740+0000 7fc3b2d3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:39.722 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.740+0000 7fc3b2d3b640 1 -- 172.21.15.112:0/3744591810 >> v1:172.21.15.112:6789/0 conn(0x7fc3ac1457f0 legacy=0x7fc3ac1d74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:39.722 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.740+0000 7fc3b2d3b640 1 -- 172.21.15.112:0/3744591810 shutdown_connections 2024-03-29T10:54:39.722 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.740+0000 7fc3b2d3b640 1 -- 172.21.15.112:0/3744591810 >> 172.21.15.112:0/3744591810 conn(0x7fc3ac066a00 msgr2=0x7fc3ac0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:39.722 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.741+0000 7fc3b2d3b640 1 -- 172.21.15.112:0/3744591810 shutdown_connections 2024-03-29T10:54:39.723 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.741+0000 7fc3b2d3b640 1 -- 172.21.15.112:0/3744591810 wait complete. 2024-03-29T10:54:39.723 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:39.741+0000 7fc3b2d3b640 1 librados: shutdown 2024-03-29T10:54:39.737 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:46.544 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:46.677 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.696+0000 7f8daef96640 10 monclient: get_monmap_and_config 2024-03-29T10:54:46.677 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.696+0000 7f8daef96640 10 monclient: build_initial_monmap 2024-03-29T10:54:46.677 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.696+0000 7f8daef96640 10 monclient: monmap: 2024-03-29T10:54:46.677 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:54:46.677 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:46.678 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:54:46.697509+0000 2024-03-29T10:54:46.678 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:54:46.697509+0000 2024-03-29T10:54:46.678 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:46.678 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:46.678 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:54:46.678 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:54:46.678 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:54:46.678 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:46.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.698+0000 7f8daef96640 1 Processor -- start 2024-03-29T10:54:46.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.698+0000 7f8daef96640 1 -- start start 2024-03-29T10:54:46.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.698+0000 7f8daef96640 10 monclient: init 2024-03-29T10:54:46.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.699+0000 7f8daef96640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:46.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.699+0000 7f8daef96640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:46.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.699+0000 7f8daef96640 10 monclient(hunting): picked mon.noname-a con 0x7f8da80b4330 addr v1:172.21.15.112:6789/0 2024-03-29T10:54:46.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.699+0000 7f8daef96640 10 monclient(hunting): picked mon.noname-c con 0x7f8da81470b0 addr v1:172.21.15.118:6789/0 2024-03-29T10:54:46.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.699+0000 7f8daef96640 10 monclient(hunting): picked mon.noname-b con 0x7f8da81439b0 addr v1:172.21.15.112:6790/0 2024-03-29T10:54:46.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.699+0000 7f8daef96640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8da806f2a0 con 0x7f8da80b4330 2024-03-29T10:54:46.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.699+0000 7f8daef96640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8da8146b90 con 0x7f8da81470b0 2024-03-29T10:54:46.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.699+0000 7f8daef96640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8da80b2ca0 con 0x7f8da81439b0 2024-03-29T10:54:46.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.699+0000 7f8daef96640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:46.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.699+0000 7f8daef96640 10 monclient(hunting): authenticate will time out at 1042.003904s 2024-03-29T10:54:46.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.699+0000 7f8dad533640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f8da81439b0 0x7f8da8151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:45256/0 (socket says 172.21.15.112:45256) 2024-03-29T10:54:46.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.699+0000 7f8dad533640 1 -- 172.21.15.112:0/2237181905 learned_addr learned my addr 172.21.15.112:0/2237181905 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:54:46.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.700+0000 7f8da77fe640 1 -- 172.21.15.112:0/2237181905 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214170292 0 0) 0x7f8da8146b90 con 0x7f8da81470b0 2024-03-29T10:54:46.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.700+0000 7f8da77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:46.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.700+0000 7f8da77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:46.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.700+0000 7f8da77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:46.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.700+0000 7f8da77fe640 10 monclient(hunting): my global_id is 4345 2024-03-29T10:54:46.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.700+0000 7f8da77fe640 1 -- 172.21.15.112:0/2237181905 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d90003520 con 0x7f8da81470b0 2024-03-29T10:54:46.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.700+0000 7f8da77fe640 1 -- 172.21.15.112:0/2237181905 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058626294 0 0) 0x7f8da80b2ca0 con 0x7f8da81439b0 2024-03-29T10:54:46.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.700+0000 7f8da77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:46.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.700+0000 7f8da77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:46.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.700+0000 7f8da77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:46.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.700+0000 7f8da77fe640 10 monclient(hunting): my global_id is 4352 2024-03-29T10:54:46.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.700+0000 7f8da77fe640 1 -- 172.21.15.112:0/2237181905 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8da8146b90 con 0x7f8da81439b0 2024-03-29T10:54:46.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.700+0000 7f8da77fe640 1 -- 172.21.15.112:0/2237181905 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4288751612 0 0) 0x7f8da8146b90 con 0x7f8da81439b0 2024-03-29T10:54:46.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.700+0000 7f8da77fe640 1 -- 172.21.15.112:0/2237181905 >> v1:172.21.15.118:6789/0 conn(0x7f8da81470b0 legacy=0x7f8da80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:46.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.700+0000 7f8da77fe640 1 -- 172.21.15.112:0/2237181905 >> v1:172.21.15.112:6789/0 conn(0x7f8da80b4330 legacy=0x7f8da80b4710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:46.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.702+0000 7f8da77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:54:46.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.702+0000 7f8da77fe640 1 monclient: found mon.noname-b 2024-03-29T10:54:46.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.702+0000 7f8da77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:46.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.702+0000 7f8da77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:54:46.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.702+0000 7f8da77fe640 1 -- 172.21.15.112:0/2237181905 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da806e0c0 con 0x7f8da81439b0 2024-03-29T10:54:46.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.702+0000 7f8da77fe640 10 monclient: _finish_auth 0 2024-03-29T10:54:46.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.702+0000 7f8da77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:54:46.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.702+0000 7f8da77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:46.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.702+0000 7f8da77fe640 1 -- 172.21.15.112:0/2237181905 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f8da00019b0 con 0x7f8da81439b0 2024-03-29T10:54:46.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.702+0000 7f8da77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:46.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.702+0000 7f8da77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:46.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.702+0000 7f8da77fe640 10 monclient: dump: 2024-03-29T10:54:46.685 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:46.685 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:46.685 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:46.685 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:46.685 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:46.685 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:46.685 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:46.685 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:46.685 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:46.685 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:46.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.702+0000 7f8daef96640 5 monclient: authenticate success, global_id 4352 2024-03-29T10:54:46.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.702+0000 7f8daef96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:46.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.703+0000 7f8da77fe640 1 -- 172.21.15.112:0/2237181905 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8da0001b80 con 0x7f8da81439b0 2024-03-29T10:54:46.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.703+0000 7f8da77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:46.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.703+0000 7f8daef96640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:46.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.703+0000 7f8daef96640 4 set_mon_vals no callback set 2024-03-29T10:54:46.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.703+0000 7f8da77fe640 1 -- 172.21.15.112:0/2237181905 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f8da0001f10 con 0x7f8da81439b0 2024-03-29T10:54:46.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.703+0000 7f8da77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:46.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.703+0000 7f8da77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:46.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.703+0000 7f8da77fe640 10 monclient: dump: 2024-03-29T10:54:46.686 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:46.686 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:46.686 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:46.686 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:46.686 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:46.686 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:46.686 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:46.687 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:46.687 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:46.687 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:46.691 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.704+0000 7f8daef96640 10 monclient: shutdown 2024-03-29T10:54:46.691 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.704+0000 7f8daef96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:46.691 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.704+0000 7f8daef96640 1 -- 172.21.15.112:0/2237181905 >> v1:172.21.15.112:6790/0 conn(0x7f8da81439b0 legacy=0x7f8da8151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:46.691 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.704+0000 7f8daef96640 1 -- 172.21.15.112:0/2237181905 shutdown_connections 2024-03-29T10:54:46.691 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.704+0000 7f8daef96640 1 -- 172.21.15.112:0/2237181905 >> 172.21.15.112:0/2237181905 conn(0x7f8da8066a00 msgr2=0x7f8da8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:46.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.704+0000 7f8daef96640 1 -- 172.21.15.112:0/2237181905 shutdown_connections 2024-03-29T10:54:46.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.704+0000 7f8daef96640 1 -- 172.21.15.112:0/2237181905 wait complete. 2024-03-29T10:54:46.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.705+0000 7f8daef96640 10 monclient: build_initial_monmap 2024-03-29T10:54:46.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.705+0000 7f8daef96640 10 monclient: monmap: 2024-03-29T10:54:46.692 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:54:46.692 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:46.692 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:54:46.692 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:54:46.692 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:46.692 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:46.692 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:54:46.692 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:54:46.692 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:54:46.693 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:46.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.705+0000 7f8daef96640 1 librados: starting msgr at 2024-03-29T10:54:46.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.705+0000 7f8daef96640 1 librados: starting objecter 2024-03-29T10:54:46.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.705+0000 7f8daef96640 1 Processor -- start 2024-03-29T10:54:46.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.705+0000 7f8daef96640 1 -- start start 2024-03-29T10:54:46.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.705+0000 7f8daef96640 1 librados: setting wanted keys 2024-03-29T10:54:46.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.705+0000 7f8daef96640 1 librados: calling monclient init 2024-03-29T10:54:46.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.705+0000 7f8daef96640 10 monclient: init 2024-03-29T10:54:46.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.706+0000 7f8daef96640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:46.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.706+0000 7f8daef96640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:54:46.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.707+0000 7f8daef96640 10 monclient(hunting): picked mon.noname-a con 0x7f8da80b4330 addr v1:172.21.15.112:6789/0 2024-03-29T10:54:46.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.707+0000 7f8daef96640 10 monclient(hunting): picked mon.noname-c con 0x7f8da81439b0 addr v1:172.21.15.112:6790/0 2024-03-29T10:54:46.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.707+0000 7f8daef96640 10 monclient(hunting): picked mon.noname-b con 0x7f8da81470b0 addr v1:172.21.15.118:6789/0 2024-03-29T10:54:46.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.707+0000 7f8daef96640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8da8126300 con 0x7f8da80b4330 2024-03-29T10:54:46.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.707+0000 7f8daef96640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8da8141000 con 0x7f8da81470b0 2024-03-29T10:54:46.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.707+0000 7f8daef96640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8da81dc2c0 con 0x7f8da81439b0 2024-03-29T10:54:46.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.707+0000 7f8daef96640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:46.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.707+0000 7f8daef96640 10 monclient(hunting): authenticate will time out at 1042.011559s 2024-03-29T10:54:46.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.707+0000 7f8dad533640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f8da81470b0 0x7f8da81408f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:51304/0 (socket says 172.21.15.112:51304) 2024-03-29T10:54:46.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.707+0000 7f8dad533640 1 -- 172.21.15.112:0/1867467356 learned_addr learned my addr 172.21.15.112:0/1867467356 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:54:46.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.707+0000 7f8da57fa640 10 client.?.objecter ms_handle_connect 0x7f8da81470b0 2024-03-29T10:54:46.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.707+0000 7f8da57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:46.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.707+0000 7f8da57fa640 10 client.?.objecter ms_handle_connect 0x7f8da81439b0 2024-03-29T10:54:46.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.707+0000 7f8da57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:46.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.707+0000 7f8da57fa640 1 -- 172.21.15.112:0/1867467356 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204384713 0 0) 0x7f8da8141000 con 0x7f8da81470b0 2024-03-29T10:54:46.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.707+0000 7f8da57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:46.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.707+0000 7f8da57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:46.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.707+0000 7f8da57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:46.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.707+0000 7f8da57fa640 10 monclient(hunting): my global_id is 4348 2024-03-29T10:54:46.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.707+0000 7f8da57fa640 1 -- 172.21.15.112:0/1867467356 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d94004eb0 con 0x7f8da81470b0 2024-03-29T10:54:46.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.708+0000 7f8da57fa640 1 -- 172.21.15.112:0/1867467356 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021701547 0 0) 0x7f8da81dc2c0 con 0x7f8da81439b0 2024-03-29T10:54:46.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.708+0000 7f8da57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:46.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.708+0000 7f8da57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:46.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.708+0000 7f8da57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:46.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.708+0000 7f8da57fa640 10 monclient(hunting): my global_id is 4355 2024-03-29T10:54:46.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.708+0000 7f8da57fa640 1 -- 172.21.15.112:0/1867467356 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8da8141000 con 0x7f8da81439b0 2024-03-29T10:54:46.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.708+0000 7f8da57fa640 1 -- 172.21.15.112:0/1867467356 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3320315144 0 0) 0x7f8da8141000 con 0x7f8da81439b0 2024-03-29T10:54:46.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.708+0000 7f8da57fa640 1 -- 172.21.15.112:0/1867467356 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8da81dc2c0 con 0x7f8da81439b0 2024-03-29T10:54:46.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.708+0000 7f8da57fa640 1 -- 172.21.15.112:0/1867467356 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f8d9c001950 con 0x7f8da81439b0 2024-03-29T10:54:46.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.708+0000 7f8da57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:46.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.708+0000 7f8da57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:46.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.708+0000 7f8da57fa640 10 monclient(hunting): dump: 2024-03-29T10:54:46.696 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:46.696 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:46.696 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:46.696 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:46.696 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:46.696 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:46.696 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:46.696 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:46.696 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:46.696 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:46.696 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.708+0000 7f8da57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:46.696 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.708+0000 7f8da57fa640 1 -- 172.21.15.112:0/1867467356 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 472372615 0 0) 0x7f8da81dc2c0 con 0x7f8da81439b0 2024-03-29T10:54:46.696 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.708+0000 7f8da57fa640 1 -- 172.21.15.112:0/1867467356 >> v1:172.21.15.118:6789/0 conn(0x7f8da81470b0 legacy=0x7f8da81408f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:46.696 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.708+0000 7f8da57fa640 1 -- 172.21.15.112:0/1867467356 >> v1:172.21.15.112:6789/0 conn(0x7f8da80b4330 legacy=0x7f8da81dbbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:54:46.696 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.709+0000 7f8da57fa640 10 monclient: _finish_hunting 0 2024-03-29T10:54:46.697 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.709+0000 7f8da57fa640 1 monclient: found mon.c 2024-03-29T10:54:46.697 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.709+0000 7f8da57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:46.697 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.709+0000 7f8da57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:54:46.697 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.709+0000 7f8da57fa640 1 -- 172.21.15.112:0/1867467356 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da81dc450 con 0x7f8da81439b0 2024-03-29T10:54:46.697 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.709+0000 7f8da57fa640 10 monclient: _finish_auth 0 2024-03-29T10:54:46.697 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.709+0000 7f8da57fa640 10 monclient: _check_auth_tickets 2024-03-29T10:54:46.697 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.709+0000 7f8da57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:46.697 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.709+0000 7f8daef96640 5 monclient: authenticate success, global_id 4355 2024-03-29T10:54:46.697 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.709+0000 7f8daef96640 10 monclient: _renew_subs 2024-03-29T10:54:46.697 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.709+0000 7f8daef96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:54:46.697 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.709+0000 7f8daef96640 1 -- 172.21.15.112:0/1867467356 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da8141770 con 0x7f8da81439b0 2024-03-29T10:54:46.697 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.709+0000 7f8daef96640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:46.697 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.709+0000 7f8daef96640 10 monclient: _renew_subs 2024-03-29T10:54:46.697 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.709+0000 7f8daef96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:54:46.697 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.709+0000 7f8daef96640 1 -- 172.21.15.112:0/1867467356 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da8126970 con 0x7f8da81439b0 2024-03-29T10:54:46.698 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.709+0000 7f8da57fa640 1 -- 172.21.15.112:0/1867467356 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d9c003610 con 0x7f8da81439b0 2024-03-29T10:54:46.699 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.709+0000 7f8daef96640 1 librados: init done 2024-03-29T10:54:46.700 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.710+0000 7f8da57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:46.700 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.710+0000 7f8da6ffd640 4 set_mon_vals no callback set 2024-03-29T10:54:46.700 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.711+0000 7f8da57fa640 1 -- 172.21.15.112:0/1867467356 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f8d9c003920 con 0x7f8da81439b0 2024-03-29T10:54:46.700 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.711+0000 7f8da57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:46.700 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.711+0000 7f8da57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:46.700 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.711+0000 7f8da57fa640 10 monclient: dump: 2024-03-29T10:54:46.700 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:46.700 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:46.700 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:46.700 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:46.700 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:46.700 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:46.700 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:46.700 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:46.701 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:46.701 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:46.701 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.711+0000 7f8da57fa640 1 -- 172.21.15.112:0/1867467356 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f8d9c003b40 con 0x7f8da81439b0 2024-03-29T10:54:46.701 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.712+0000 7f8da57fa640 1 -- 172.21.15.112:0/1867467356 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f8d9c0838e0 con 0x7f8da81439b0 2024-03-29T10:54:46.701 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.712+0000 7f8da57fa640 10 client.4355.objecter ms_dispatch 0x7f8da8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:46.701 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.712+0000 7f8da57fa640 3 client.4355.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:46.701 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.712+0000 7f8da57fa640 3 client.4355.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:46.701 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.712+0000 7f8dacd32640 10 monclient: get_auth_request con 0x7f8d9406dd60 auth_method 0 2024-03-29T10:54:46.701 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.712+0000 7f8da57fa640 20 client.4355.objecter dump_active .. 0 homeless 2024-03-29T10:54:46.701 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.712+0000 7f8da57fa640 10 client.4355.objecter ms_handle_connect 0x7f8d9406dd60 2024-03-29T10:54:46.701 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.712+0000 7f8daef96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:46.701 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.712+0000 7f8daef96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:46.701 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.712+0000 7f8daef96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:54:46.701 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.712+0000 7f8daef96640 1 -- 172.21.15.112:0/1867467356 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d88005080 con 0x7f8da81439b0 2024-03-29T10:54:46.702 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.721+0000 7f8da57fa640 1 -- 172.21.15.112:0/1867467356 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8d9c04f240 con 0x7f8da81439b0 2024-03-29T10:54:46.702 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.721+0000 7f8da57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:46.702 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:46.721+0000 7f8da57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:47.007 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:47.026+0000 7f8daef96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:47.008 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:47.026+0000 7f8daef96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:47.008 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:47.026+0000 7f8daef96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:54:47.008 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:47.026+0000 7f8daef96640 1 -- 172.21.15.112:0/1867467356 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d88005330 con 0x7f8da81439b0 2024-03-29T10:54:47.008 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:47.027+0000 7f8da57fa640 1 -- 172.21.15.112:0/1867467356 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f8d9c051170 con 0x7f8da81439b0 2024-03-29T10:54:47.009 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:47.027+0000 7f8da57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:47.009 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:47.027+0000 7f8da57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:47.009 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:54:47.009 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:47.012 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:47.030+0000 7f8daef96640 10 librados: watch_flush enter 2024-03-29T10:54:47.012 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:47.030+0000 7f8daef96640 10 librados: watch_flush exit 2024-03-29T10:54:47.012 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:47.030+0000 7f8daef96640 20 client.4355.objecter shutdown clearing up homeless session... 2024-03-29T10:54:47.012 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:47.030+0000 7f8daef96640 10 client.4355.objecter successfully canceled tick 2024-03-29T10:54:47.013 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:47.030+0000 7f8daef96640 1 -- 172.21.15.112:0/1867467356 >> v1:172.21.15.118:6816/3181073408 conn(0x7f8d9406dd60 legacy=0x7f8d94070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:47.013 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:47.030+0000 7f8daef96640 10 monclient: shutdown 2024-03-29T10:54:47.013 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:47.030+0000 7f8daef96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:47.013 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:47.030+0000 7f8daef96640 1 -- 172.21.15.112:0/1867467356 >> v1:172.21.15.112:6790/0 conn(0x7f8da81439b0 legacy=0x7f8da81401e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:47.013 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:47.031+0000 7f8daef96640 1 -- 172.21.15.112:0/1867467356 shutdown_connections 2024-03-29T10:54:47.013 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:47.031+0000 7f8daef96640 1 -- 172.21.15.112:0/1867467356 >> 172.21.15.112:0/1867467356 conn(0x7f8da8066a00 msgr2=0x7f8da80b4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:47.013 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:47.031+0000 7f8daef96640 1 -- 172.21.15.112:0/1867467356 shutdown_connections 2024-03-29T10:54:47.013 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:47.031+0000 7f8daef96640 1 -- 172.21.15.112:0/1867467356 wait complete. 2024-03-29T10:54:47.013 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:47.031+0000 7f8daef96640 1 librados: shutdown 2024-03-29T10:54:47.025 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:53.833 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:53.980 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:53.999+0000 7f32fad1c640 10 monclient: get_monmap_and_config 2024-03-29T10:54:53.980 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:53.999+0000 7f32fad1c640 10 monclient: build_initial_monmap 2024-03-29T10:54:53.980 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:53.999+0000 7f32fad1c640 10 monclient: monmap: 2024-03-29T10:54:53.980 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:54:53.980 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:53.980 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:54:54.000372+0000 2024-03-29T10:54:53.981 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:54:54.000372+0000 2024-03-29T10:54:53.981 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:53.981 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:53.981 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:54:53.981 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:54:53.981 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:54:53.981 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:53.981 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.000+0000 7f32fad1c640 1 Processor -- start 2024-03-29T10:54:53.982 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.000+0000 7f32fad1c640 1 -- start start 2024-03-29T10:54:53.982 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.000+0000 7f32fad1c640 10 monclient: init 2024-03-29T10:54:53.982 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.001+0000 7f32fad1c640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:53.982 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.001+0000 7f32fad1c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:53.982 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.001+0000 7f32fad1c640 10 monclient(hunting): picked mon.noname-b con 0x7f32f40b0770 addr v1:172.21.15.112:6790/0 2024-03-29T10:54:53.982 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.001+0000 7f32fad1c640 10 monclient(hunting): picked mon.noname-a con 0x7f32f414a700 addr v1:172.21.15.112:6789/0 2024-03-29T10:54:53.982 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.001+0000 7f32fad1c640 10 monclient(hunting): picked mon.noname-c con 0x7f32f40ac5a0 addr v1:172.21.15.118:6789/0 2024-03-29T10:54:53.983 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.001+0000 7f32fad1c640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f40aba70 con 0x7f32f414a700 2024-03-29T10:54:53.983 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.001+0000 7f32fad1c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f40aecb0 con 0x7f32f40ac5a0 2024-03-29T10:54:53.983 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.001+0000 7f32fad1c640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f40af150 con 0x7f32f40b0770 2024-03-29T10:54:53.983 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.001+0000 7f32fad1c640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:53.983 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.001+0000 7f32fad1c640 10 monclient(hunting): authenticate will time out at 1049.305748s 2024-03-29T10:54:53.983 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.002+0000 7f32f8ab8640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f32f40b0770 0x7f32f4149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:49476/0 (socket says 172.21.15.112:49476) 2024-03-29T10:54:53.983 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.003+0000 7f32f8ab8640 1 -- 172.21.15.112:0/2976929588 learned_addr learned my addr 172.21.15.112:0/2976929588 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:54:53.984 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.003+0000 7f32f37fe640 1 -- 172.21.15.112:0/2976929588 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3784564982 0 0) 0x7f32f40af150 con 0x7f32f40b0770 2024-03-29T10:54:53.984 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.003+0000 7f32f37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:53.984 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.003+0000 7f32f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:53.984 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.003+0000 7f32f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:53.984 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.003+0000 7f32f37fe640 10 monclient(hunting): my global_id is 4358 2024-03-29T10:54:53.984 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.003+0000 7f32f37fe640 1 -- 172.21.15.112:0/2976929588 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32e4003520 con 0x7f32f40b0770 2024-03-29T10:54:53.985 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.004+0000 7f32f37fe640 1 -- 172.21.15.112:0/2976929588 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3650570522 0 0) 0x7f32e4003520 con 0x7f32f40b0770 2024-03-29T10:54:53.985 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.004+0000 7f32f37fe640 1 -- 172.21.15.112:0/2976929588 >> v1:172.21.15.118:6789/0 conn(0x7f32f40ac5a0 legacy=0x7f32f4151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:54:53.985 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.004+0000 7f32f37fe640 1 -- 172.21.15.112:0/2976929588 >> v1:172.21.15.112:6789/0 conn(0x7f32f414a700 legacy=0x7f32f40abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:53.990 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.007+0000 7f32f37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:54:53.990 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.007+0000 7f32f37fe640 1 monclient: found mon.noname-b 2024-03-29T10:54:53.990 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.007+0000 7f32f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:53.990 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.007+0000 7f32f37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:54:53.990 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.007+0000 7f32f37fe640 1 -- 172.21.15.112:0/2976929588 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f40ac9d0 con 0x7f32f40b0770 2024-03-29T10:54:53.990 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.007+0000 7f32f37fe640 10 monclient: _finish_auth 0 2024-03-29T10:54:53.991 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.007+0000 7f32f37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:54:53.991 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.008+0000 7f32f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:53.991 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.008+0000 7f32fad1c640 5 monclient: authenticate success, global_id 4358 2024-03-29T10:54:53.991 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.008+0000 7f32fad1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:53.991 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.008+0000 7f32f37fe640 1 -- 172.21.15.112:0/2976929588 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f32e8001690 con 0x7f32f40b0770 2024-03-29T10:54:53.991 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.008+0000 7f32f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:53.991 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.008+0000 7f32f37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:53.991 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.008+0000 7f32f37fe640 10 monclient: dump: 2024-03-29T10:54:53.991 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:53.991 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:53.991 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:53.991 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:53.991 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:53.991 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:53.991 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:53.992 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:53.992 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:53.992 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:53.992 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.008+0000 7f32fad1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:53.992 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.008+0000 7f32f37fe640 1 -- 172.21.15.112:0/2976929588 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32e8001920 con 0x7f32f40b0770 2024-03-29T10:54:53.992 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.008+0000 7f32f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:53.992 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.008+0000 7f32fad1c640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:53.992 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.008+0000 7f32fad1c640 4 set_mon_vals no callback set 2024-03-29T10:54:53.992 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.008+0000 7f32f37fe640 1 -- 172.21.15.112:0/2976929588 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f32e8001cb0 con 0x7f32f40b0770 2024-03-29T10:54:53.992 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.008+0000 7f32f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:53.992 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.008+0000 7f32f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:53.992 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.008+0000 7f32f37fe640 10 monclient: dump: 2024-03-29T10:54:53.992 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:53.992 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:53.993 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:53.993 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:53.993 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:53.993 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:53.993 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:53.993 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:53.993 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:53.993 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:53.993 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.009+0000 7f32fad1c640 10 monclient: shutdown 2024-03-29T10:54:53.993 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.009+0000 7f32fad1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:53.993 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.009+0000 7f32fad1c640 1 -- 172.21.15.112:0/2976929588 >> v1:172.21.15.112:6790/0 conn(0x7f32f40b0770 legacy=0x7f32f4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:53.993 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.009+0000 7f32fad1c640 1 -- 172.21.15.112:0/2976929588 shutdown_connections 2024-03-29T10:54:53.993 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.009+0000 7f32fad1c640 1 -- 172.21.15.112:0/2976929588 >> 172.21.15.112:0/2976929588 conn(0x7f32f406b660 msgr2=0x7f32f406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:53.993 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.009+0000 7f32fad1c640 1 -- 172.21.15.112:0/2976929588 shutdown_connections 2024-03-29T10:54:54.002 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.009+0000 7f32fad1c640 1 -- 172.21.15.112:0/2976929588 wait complete. 2024-03-29T10:54:54.002 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.010+0000 7f32fad1c640 10 monclient: build_initial_monmap 2024-03-29T10:54:54.002 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.010+0000 7f32fad1c640 10 monclient: monmap: 2024-03-29T10:54:54.002 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:54:54.002 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:54.002 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:54:54.002 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:54:54.002 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:54.002 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:54.003 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:54:54.003 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:54:54.003 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:54:54.003 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:54.003 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.011+0000 7f32fad1c640 1 librados: starting msgr at 2024-03-29T10:54:54.003 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.011+0000 7f32fad1c640 1 librados: starting objecter 2024-03-29T10:54:54.003 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.011+0000 7f32fad1c640 1 Processor -- start 2024-03-29T10:54:54.003 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.011+0000 7f32fad1c640 1 -- start start 2024-03-29T10:54:54.003 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.011+0000 7f32fad1c640 1 librados: setting wanted keys 2024-03-29T10:54:54.003 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.011+0000 7f32fad1c640 1 librados: calling monclient init 2024-03-29T10:54:54.003 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.011+0000 7f32fad1c640 10 monclient: init 2024-03-29T10:54:54.003 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.012+0000 7f32fad1c640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:54.003 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.012+0000 7f32fad1c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:54:54.003 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.012+0000 7f32fad1c640 10 monclient(hunting): picked mon.noname-c con 0x7f32f40ac5a0 addr v1:172.21.15.112:6790/0 2024-03-29T10:54:54.004 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.012+0000 7f32fad1c640 10 monclient(hunting): picked mon.noname-b con 0x7f32f40b0770 addr v1:172.21.15.118:6789/0 2024-03-29T10:54:54.004 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.012+0000 7f32fad1c640 10 monclient(hunting): picked mon.noname-a con 0x7f32f414a700 addr v1:172.21.15.112:6789/0 2024-03-29T10:54:54.004 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.012+0000 7f32fad1c640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f41dbe40 con 0x7f32f414a700 2024-03-29T10:54:54.004 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.012+0000 7f32fad1c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f4145250 con 0x7f32f40b0770 2024-03-29T10:54:54.004 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.012+0000 7f32fad1c640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f41453e0 con 0x7f32f40ac5a0 2024-03-29T10:54:54.004 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.012+0000 7f32fad1c640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:54.004 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.012+0000 7f32fad1c640 10 monclient(hunting): authenticate will time out at 1049.316590s 2024-03-29T10:54:54.004 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.012+0000 7f32f8ab8640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f32f40ac5a0 0x7f32f4144b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:49616/0 (socket says 172.21.15.112:49616) 2024-03-29T10:54:54.004 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.012+0000 7f32f8ab8640 1 -- 172.21.15.112:0/388713818 learned_addr learned my addr 172.21.15.112:0/388713818 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:54:54.004 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.012+0000 7f32f17fa640 10 client.?.objecter ms_handle_connect 0x7f32f40ac5a0 2024-03-29T10:54:54.004 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.012+0000 7f32f17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:54.004 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.013+0000 7f32f17fa640 1 -- 172.21.15.112:0/388713818 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623564482 0 0) 0x7f32f41453e0 con 0x7f32f40ac5a0 2024-03-29T10:54:54.004 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.013+0000 7f32f17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:54.004 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.013+0000 7f32f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:54.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.013+0000 7f32f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:54.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.013+0000 7f32f17fa640 10 monclient(hunting): my global_id is 4361 2024-03-29T10:54:54.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.013+0000 7f32f17fa640 1 -- 172.21.15.112:0/388713818 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32d8004eb0 con 0x7f32f40ac5a0 2024-03-29T10:54:54.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.014+0000 7f32f17fa640 1 -- 172.21.15.112:0/388713818 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3461523784 0 0) 0x7f32d8004eb0 con 0x7f32f40ac5a0 2024-03-29T10:54:54.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.014+0000 7f32f17fa640 1 -- 172.21.15.112:0/388713818 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32f41453e0 con 0x7f32f40ac5a0 2024-03-29T10:54:54.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.014+0000 7f32f17fa640 1 -- 172.21.15.112:0/388713818 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f32e8001610 con 0x7f32f40ac5a0 2024-03-29T10:54:54.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.014+0000 7f32f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:54.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.014+0000 7f32f17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:54.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.014+0000 7f32f17fa640 10 monclient(hunting): dump: 2024-03-29T10:54:54.005 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:54.005 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:54.005 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:54.005 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:54.005 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:54.006 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:54.006 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:54.006 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:54.006 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:54.006 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:54.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.014+0000 7f32f17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:54.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.015+0000 7f32f17fa640 1 -- 172.21.15.112:0/388713818 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2189974163 0 0) 0x7f32f41453e0 con 0x7f32f40ac5a0 2024-03-29T10:54:54.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.015+0000 7f32f17fa640 1 -- 172.21.15.112:0/388713818 >> v1:172.21.15.118:6789/0 conn(0x7f32f40b0770 legacy=0x7f32f41db1d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:54:54.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.015+0000 7f32f17fa640 1 -- 172.21.15.112:0/388713818 >> v1:172.21.15.112:6789/0 conn(0x7f32f414a700 legacy=0x7f32f41de7d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:54:54.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.017+0000 7f32f17fa640 10 monclient: _finish_hunting 0 2024-03-29T10:54:54.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.017+0000 7f32f17fa640 1 monclient: found mon.c 2024-03-29T10:54:54.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.017+0000 7f32f17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:54.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.017+0000 7f32f17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:54:54.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.017+0000 7f32f17fa640 1 -- 172.21.15.112:0/388713818 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f41db8e0 con 0x7f32f40ac5a0 2024-03-29T10:54:54.007 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.017+0000 7f32f17fa640 10 monclient: _finish_auth 0 2024-03-29T10:54:54.007 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.017+0000 7f32f17fa640 10 monclient: _check_auth_tickets 2024-03-29T10:54:54.007 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.017+0000 7f32f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:54.007 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.017+0000 7f32fad1c640 5 monclient: authenticate success, global_id 4361 2024-03-29T10:54:54.007 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.017+0000 7f32fad1c640 10 monclient: _renew_subs 2024-03-29T10:54:54.007 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.017+0000 7f32fad1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:54:54.007 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.017+0000 7f32fad1c640 1 -- 172.21.15.112:0/388713818 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f41dbb90 con 0x7f32f40ac5a0 2024-03-29T10:54:54.007 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.017+0000 7f32fad1c640 10 client.4361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:54.007 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.017+0000 7f32fad1c640 10 monclient: _renew_subs 2024-03-29T10:54:54.007 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.017+0000 7f32fad1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:54:54.007 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.017+0000 7f32fad1c640 1 -- 172.21.15.112:0/388713818 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f4070d60 con 0x7f32f40ac5a0 2024-03-29T10:54:54.007 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.017+0000 7f32fad1c640 1 librados: init done 2024-03-29T10:54:54.007 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.019+0000 7f32deffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:54.007 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.019+0000 7f32deffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:54.007 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.019+0000 7f32deffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:54:54.008 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.019+0000 7f32deffd640 1 -- 172.21.15.112:0/388713818 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f41dc020 con 0x7f32f40ac5a0 2024-03-29T10:54:54.008 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.023+0000 7f32f17fa640 1 -- 172.21.15.112:0/388713818 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32e8004bf0 con 0x7f32f40ac5a0 2024-03-29T10:54:54.008 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.023+0000 7f32f17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:54.008 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.023+0000 7f32f17fa640 1 -- 172.21.15.112:0/388713818 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f32e8001770 con 0x7f32f40ac5a0 2024-03-29T10:54:54.009 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.023+0000 7f32f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:54.009 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.023+0000 7f32f17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:54.009 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.023+0000 7f32f17fa640 10 monclient: dump: 2024-03-29T10:54:54.009 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:54:54.009 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:54:54.009 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:54.009 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:54:54.009 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:54.009 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:54:54.009 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:54:54.009 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:54:54.009 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:54:54.009 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:54:54.009 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.023+0000 7f32f17fa640 1 -- 172.21.15.112:0/388713818 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f32e801adb0 con 0x7f32f40ac5a0 2024-03-29T10:54:54.010 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.023+0000 7f32f2ffd640 4 set_mon_vals no callback set 2024-03-29T10:54:54.010 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.023+0000 7f32f3fff640 10 monclient: get_auth_request con 0x7f32d806dd10 auth_method 0 2024-03-29T10:54:54.010 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.024+0000 7f32f17fa640 1 -- 172.21.15.112:0/388713818 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f32e8084de0 con 0x7f32f40ac5a0 2024-03-29T10:54:54.010 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.024+0000 7f32f17fa640 10 client.4361.objecter ms_dispatch 0x7f32f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:54.010 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.024+0000 7f32f17fa640 3 client.4361.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:54.010 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.024+0000 7f32f17fa640 3 client.4361.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:54.010 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.024+0000 7f32f17fa640 20 client.4361.objecter dump_active .. 0 homeless 2024-03-29T10:54:54.010 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.024+0000 7f32f17fa640 10 client.4361.objecter ms_handle_connect 0x7f32d806dd10 2024-03-29T10:54:54.011 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.031+0000 7f32f17fa640 1 -- 172.21.15.112:0/388713818 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f32e8050740 con 0x7f32f40ac5a0 2024-03-29T10:54:54.011 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.031+0000 7f32f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:54.012 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.031+0000 7f32f17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:54.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.360+0000 7f32deffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:54.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.360+0000 7f32deffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:54.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.360+0000 7f32deffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:54:54.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.360+0000 7f32deffd640 1 -- 172.21.15.112:0/388713818 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32f4075f10 con 0x7f32f40ac5a0 2024-03-29T10:54:54.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.361+0000 7f32f17fa640 1 -- 172.21.15.112:0/388713818 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f32e8052670 con 0x7f32f40ac5a0 2024-03-29T10:54:54.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.361+0000 7f32f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:54.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.361+0000 7f32f17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:54.343 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:54:54.344 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:54.347 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.365+0000 7f32deffd640 10 librados: watch_flush enter 2024-03-29T10:54:54.347 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.365+0000 7f32deffd640 10 librados: watch_flush exit 2024-03-29T10:54:54.347 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.365+0000 7f32deffd640 20 client.4361.objecter shutdown clearing up homeless session... 2024-03-29T10:54:54.347 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.365+0000 7f32deffd640 10 client.4361.objecter successfully canceled tick 2024-03-29T10:54:54.348 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.365+0000 7f32deffd640 1 -- 172.21.15.112:0/388713818 >> v1:172.21.15.118:6816/3181073408 conn(0x7f32d806dd10 legacy=0x7f32d80701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:54.348 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.365+0000 7f32deffd640 10 monclient: shutdown 2024-03-29T10:54:54.348 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.365+0000 7f32deffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:54.348 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.366+0000 7f32deffd640 1 -- 172.21.15.112:0/388713818 >> v1:172.21.15.112:6790/0 conn(0x7f32f40ac5a0 legacy=0x7f32f4144b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:54.348 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.366+0000 7f32deffd640 1 -- 172.21.15.112:0/388713818 shutdown_connections 2024-03-29T10:54:54.348 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.366+0000 7f32deffd640 1 -- 172.21.15.112:0/388713818 >> 172.21.15.112:0/388713818 conn(0x7f32f406b660 msgr2=0x7f32f406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:54.349 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.368+0000 7f32deffd640 1 -- 172.21.15.112:0/388713818 shutdown_connections 2024-03-29T10:54:54.349 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.368+0000 7f32deffd640 1 -- 172.21.15.112:0/388713818 wait complete. 2024-03-29T10:54:54.349 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:54:54.368+0000 7f32deffd640 1 librados: shutdown 2024-03-29T10:54:54.367 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:01.174 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:01.324 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.343+0000 7fe5ee7a4640 10 monclient: get_monmap_and_config 2024-03-29T10:55:01.324 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.343+0000 7fe5ee7a4640 10 monclient: build_initial_monmap 2024-03-29T10:55:01.324 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.343+0000 7fe5ee7a4640 10 monclient: monmap: 2024-03-29T10:55:01.324 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:55:01.324 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:01.324 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:55:01.344217+0000 2024-03-29T10:55:01.324 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:55:01.344217+0000 2024-03-29T10:55:01.324 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:01.325 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:01.325 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:55:01.325 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:55:01.325 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:55:01.325 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:01.325 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.344+0000 7fe5ee7a4640 1 Processor -- start 2024-03-29T10:55:01.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.344+0000 7fe5ee7a4640 1 -- start start 2024-03-29T10:55:01.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.344+0000 7fe5ee7a4640 10 monclient: init 2024-03-29T10:55:01.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.345+0000 7fe5ee7a4640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:01.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.345+0000 7fe5ee7a4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:55:01.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.346+0000 7fe5ee7a4640 10 monclient(hunting): picked mon.noname-c con 0x7fe5e81457f0 addr v1:172.21.15.118:6789/0 2024-03-29T10:55:01.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.346+0000 7fe5ee7a4640 10 monclient(hunting): picked mon.noname-b con 0x7fe5e813f650 addr v1:172.21.15.112:6790/0 2024-03-29T10:55:01.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.346+0000 7fe5ee7a4640 10 monclient(hunting): picked mon.noname-a con 0x7fe5e80a9d70 addr v1:172.21.15.112:6789/0 2024-03-29T10:55:01.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.346+0000 7fe5ee7a4640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e80aaef0 con 0x7fe5e80a9d70 2024-03-29T10:55:01.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.346+0000 7fe5ee7a4640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e8145bd0 con 0x7fe5e81457f0 2024-03-29T10:55:01.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.346+0000 7fe5ee7a4640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e8145eb0 con 0x7fe5e813f650 2024-03-29T10:55:01.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.346+0000 7fe5ee7a4640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:01.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.346+0000 7fe5ee7a4640 10 monclient(hunting): authenticate will time out at 1056.651022s 2024-03-29T10:55:01.329 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.349+0000 7fe5e77fe640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe5e813f650 0x7fe5e80b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:57832/0 (socket says 172.21.15.112:57832) 2024-03-29T10:55:01.329 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.349+0000 7fe5e77fe640 1 -- 172.21.15.112:0/3963819129 learned_addr learned my addr 172.21.15.112:0/3963819129 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:55:01.330 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.349+0000 7fe5e6ffd640 1 -- 172.21.15.112:0/3963819129 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013489030 0 0) 0x7fe5e8145eb0 con 0x7fe5e813f650 2024-03-29T10:55:01.330 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.349+0000 7fe5e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:01.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.349+0000 7fe5e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:01.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.349+0000 7fe5e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:01.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.349+0000 7fe5e6ffd640 10 monclient(hunting): my global_id is 4364 2024-03-29T10:55:01.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.349+0000 7fe5e6ffd640 1 -- 172.21.15.112:0/3963819129 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5d0003520 con 0x7fe5e813f650 2024-03-29T10:55:01.334 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.350+0000 7fe5e6ffd640 1 -- 172.21.15.112:0/3963819129 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4138342511 0 0) 0x7fe5d0003520 con 0x7fe5e813f650 2024-03-29T10:55:01.334 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.350+0000 7fe5e6ffd640 1 -- 172.21.15.112:0/3963819129 >> v1:172.21.15.118:6789/0 conn(0x7fe5e81457f0 legacy=0x7fe5e813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:01.334 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.350+0000 7fe5e6ffd640 1 -- 172.21.15.112:0/3963819129 >> v1:172.21.15.112:6789/0 conn(0x7fe5e80a9d70 legacy=0x7fe5e814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:55:01.334 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.350+0000 7fe5e6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:55:01.334 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.350+0000 7fe5e6ffd640 1 monclient: found mon.noname-b 2024-03-29T10:55:01.334 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.350+0000 7fe5e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:01.334 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.350+0000 7fe5e6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:55:01.334 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.350+0000 7fe5e6ffd640 1 -- 172.21.15.112:0/3963819129 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e8146340 con 0x7fe5e813f650 2024-03-29T10:55:01.334 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.350+0000 7fe5e6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:55:01.335 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.350+0000 7fe5e6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:55:01.335 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.350+0000 7fe5e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:01.335 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.350+0000 7fe5e6ffd640 1 -- 172.21.15.112:0/3963819129 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe5dc001690 con 0x7fe5e813f650 2024-03-29T10:55:01.335 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.350+0000 7fe5e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:01.335 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.350+0000 7fe5e6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:01.335 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.350+0000 7fe5e6ffd640 10 monclient: dump: 2024-03-29T10:55:01.335 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:01.335 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:01.335 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:01.335 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:01.335 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:01.335 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:01.335 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:01.335 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:01.335 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:01.336 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:01.336 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.350+0000 7fe5ee7a4640 5 monclient: authenticate success, global_id 4364 2024-03-29T10:55:01.336 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.350+0000 7fe5ee7a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:01.336 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.351+0000 7fe5e6ffd640 1 -- 172.21.15.112:0/3963819129 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5dc001920 con 0x7fe5e813f650 2024-03-29T10:55:01.336 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.351+0000 7fe5e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:01.336 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.351+0000 7fe5e6ffd640 1 -- 172.21.15.112:0/3963819129 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe5dc001cb0 con 0x7fe5e813f650 2024-03-29T10:55:01.336 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.351+0000 7fe5e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:01.336 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.351+0000 7fe5e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:01.336 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.351+0000 7fe5e6ffd640 10 monclient: dump: 2024-03-29T10:55:01.336 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:01.336 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:01.336 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:01.336 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:01.336 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:01.337 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:01.337 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:01.337 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:01.337 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:01.337 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:01.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.351+0000 7fe5ee7a4640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:01.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.351+0000 7fe5ee7a4640 4 set_mon_vals no callback set 2024-03-29T10:55:01.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.352+0000 7fe5ee7a4640 10 monclient: shutdown 2024-03-29T10:55:01.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.352+0000 7fe5ee7a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:01.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.352+0000 7fe5ee7a4640 1 -- 172.21.15.112:0/3963819129 >> v1:172.21.15.112:6790/0 conn(0x7fe5e813f650 legacy=0x7fe5e80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:01.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.353+0000 7fe5ee7a4640 1 -- 172.21.15.112:0/3963819129 shutdown_connections 2024-03-29T10:55:01.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.353+0000 7fe5ee7a4640 1 -- 172.21.15.112:0/3963819129 >> 172.21.15.112:0/3963819129 conn(0x7fe5e8066a00 msgr2=0x7fe5e8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:01.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.353+0000 7fe5ee7a4640 1 -- 172.21.15.112:0/3963819129 shutdown_connections 2024-03-29T10:55:01.339 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.354+0000 7fe5ee7a4640 1 -- 172.21.15.112:0/3963819129 wait complete. 2024-03-29T10:55:01.339 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.355+0000 7fe5ee7a4640 10 monclient: build_initial_monmap 2024-03-29T10:55:01.339 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.355+0000 7fe5ee7a4640 10 monclient: monmap: 2024-03-29T10:55:01.339 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:55:01.339 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:01.339 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:55:01.339 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:55:01.340 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:01.340 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:01.340 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:55:01.340 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:55:01.340 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:55:01.340 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:01.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.355+0000 7fe5ee7a4640 1 librados: starting msgr at 2024-03-29T10:55:01.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.355+0000 7fe5ee7a4640 1 librados: starting objecter 2024-03-29T10:55:01.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.356+0000 7fe5ee7a4640 1 Processor -- start 2024-03-29T10:55:01.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.356+0000 7fe5ee7a4640 1 -- start start 2024-03-29T10:55:01.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.356+0000 7fe5ee7a4640 1 librados: setting wanted keys 2024-03-29T10:55:01.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.356+0000 7fe5ee7a4640 1 librados: calling monclient init 2024-03-29T10:55:01.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.356+0000 7fe5ee7a4640 10 monclient: init 2024-03-29T10:55:01.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.357+0000 7fe5ee7a4640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:01.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.357+0000 7fe5ee7a4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:55:01.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.357+0000 7fe5ee7a4640 10 monclient(hunting): picked mon.noname-b con 0x7fe5e80a9d70 addr v1:172.21.15.118:6789/0 2024-03-29T10:55:01.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.357+0000 7fe5ee7a4640 10 monclient(hunting): picked mon.noname-c con 0x7fe5e813f650 addr v1:172.21.15.112:6790/0 2024-03-29T10:55:01.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.357+0000 7fe5e77fe640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe5e813f650 0x7fe5e8136c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:58096/0 (socket says 172.21.15.112:58096) 2024-03-29T10:55:01.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.357+0000 7fe5e77fe640 1 -- 172.21.15.112:0/147016070 learned_addr learned my addr 172.21.15.112:0/147016070 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:55:01.346 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.358+0000 7fe5e4ff9640 10 client.?.objecter ms_handle_connect 0x7fe5e813f650 2024-03-29T10:55:01.346 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.360+0000 7fe5e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:01.346 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.360+0000 7fe5ee7a4640 10 monclient(hunting): picked mon.noname-a con 0x7fe5e81457f0 addr v1:172.21.15.112:6789/0 2024-03-29T10:55:01.346 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.361+0000 7fe5ee7a4640 1 -- 172.21.15.112:0/147016070 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e81420a0 con 0x7fe5e81457f0 2024-03-29T10:55:01.346 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.361+0000 7fe5e4ff9640 10 client.?.objecter ms_handle_connect 0x7fe5e81457f0 2024-03-29T10:55:01.346 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.361+0000 7fe5e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:01.347 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.361+0000 7fe5ee7a4640 1 -- 172.21.15.112:0/147016070 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e8137a20 con 0x7fe5e80a9d70 2024-03-29T10:55:01.347 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.361+0000 7fe5ee7a4640 1 -- 172.21.15.112:0/147016070 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e80ac5f0 con 0x7fe5e813f650 2024-03-29T10:55:01.347 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.361+0000 7fe5e4ff9640 1 -- 172.21.15.112:0/147016070 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3158728518 0 0) 0x7fe5e81420a0 con 0x7fe5e81457f0 2024-03-29T10:55:01.347 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.361+0000 7fe5ee7a4640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:01.347 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.361+0000 7fe5ee7a4640 10 monclient(hunting): authenticate will time out at 1056.665788s 2024-03-29T10:55:01.347 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.361+0000 7fe5e4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:55:01.347 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.361+0000 7fe5e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:01.347 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.361+0000 7fe5e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:01.347 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.361+0000 7fe5e4ff9640 10 monclient(hunting): my global_id is 4353 2024-03-29T10:55:01.347 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.361+0000 7fe5e4ff9640 1 -- 172.21.15.112:0/147016070 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5d4004eb0 con 0x7fe5e81457f0 2024-03-29T10:55:01.347 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.362+0000 7fe5e4ff9640 1 -- 172.21.15.112:0/147016070 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618672110 0 0) 0x7fe5e80ac5f0 con 0x7fe5e813f650 2024-03-29T10:55:01.347 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.362+0000 7fe5e4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:55:01.347 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.362+0000 7fe5e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:01.347 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.362+0000 7fe5e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:01.348 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.362+0000 7fe5e4ff9640 10 monclient(hunting): my global_id is 4367 2024-03-29T10:55:01.348 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.362+0000 7fe5e4ff9640 1 -- 172.21.15.112:0/147016070 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5e81420a0 con 0x7fe5e813f650 2024-03-29T10:55:01.348 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.362+0000 7fe5e4ff9640 1 -- 172.21.15.112:0/147016070 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1595427683 0 0) 0x7fe5d4004eb0 con 0x7fe5e81457f0 2024-03-29T10:55:01.348 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.362+0000 7fe5e4ff9640 1 -- 172.21.15.112:0/147016070 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5e80ac5f0 con 0x7fe5e81457f0 2024-03-29T10:55:01.348 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.362+0000 7fe5e4ff9640 1 -- 172.21.15.112:0/147016070 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe5e0001b70 con 0x7fe5e81457f0 2024-03-29T10:55:01.348 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.362+0000 7fe5e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:01.348 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.362+0000 7fe5e4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:01.348 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.362+0000 7fe5e4ff9640 10 monclient(hunting): dump: 2024-03-29T10:55:01.348 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:01.348 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:01.348 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:01.348 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:01.348 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:01.348 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:01.349 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:01.349 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:01.349 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:01.349 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:01.349 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.363+0000 7fe5e4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:01.349 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.363+0000 7fe5e4ff9640 1 -- 172.21.15.112:0/147016070 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2531526017 0 0) 0x7fe5e80ac5f0 con 0x7fe5e81457f0 2024-03-29T10:55:01.349 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.363+0000 7fe5e4ff9640 1 -- 172.21.15.112:0/147016070 >> v1:172.21.15.112:6790/0 conn(0x7fe5e813f650 legacy=0x7fe5e8136c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:01.349 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.363+0000 7fe5e4ff9640 1 -- 172.21.15.112:0/147016070 >> v1:172.21.15.118:6789/0 conn(0x7fe5e80a9d70 legacy=0x7fe5e8142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:01.349 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.363+0000 7fe5e4ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:55:01.349 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.363+0000 7fe5e4ff9640 1 monclient: found mon.a 2024-03-29T10:55:01.349 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.363+0000 7fe5e4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:01.349 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.363+0000 7fe5e4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:55:01.349 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.363+0000 7fe5e4ff9640 1 -- 172.21.15.112:0/147016070 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e80ac8d0 con 0x7fe5e81457f0 2024-03-29T10:55:01.349 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.363+0000 7fe5e4ff9640 10 monclient: _finish_auth 0 2024-03-29T10:55:01.349 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.363+0000 7fe5e4ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:55:01.350 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.363+0000 7fe5e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:01.350 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.363+0000 7fe5e4ff9640 10 client.?.objecter ms_handle_connect 0x7fe5e80a9d70 2024-03-29T10:55:01.350 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.363+0000 7fe5ee7a4640 5 monclient: authenticate success, global_id 4353 2024-03-29T10:55:01.350 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.363+0000 7fe5ee7a4640 10 monclient: _renew_subs 2024-03-29T10:55:01.350 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.364+0000 7fe5ee7a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:55:01.350 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.364+0000 7fe5ee7a4640 1 -- 172.21.15.112:0/147016070 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e8142c40 con 0x7fe5e81457f0 2024-03-29T10:55:01.350 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.364+0000 7fe5ee7a4640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:01.350 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.364+0000 7fe5ee7a4640 10 monclient: _renew_subs 2024-03-29T10:55:01.350 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.364+0000 7fe5ee7a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:55:01.350 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.364+0000 7fe5ee7a4640 1 -- 172.21.15.112:0/147016070 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e813fda0 con 0x7fe5e81457f0 2024-03-29T10:55:01.350 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.364+0000 7fe5ee7a4640 1 librados: init done 2024-03-29T10:55:01.350 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.364+0000 7fe5e4ff9640 10 client.4353.objecter resend_mon_ops 2024-03-29T10:55:01.350 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.364+0000 7fe5e4ff9640 1 -- 172.21.15.112:0/147016070 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5e0001cd0 con 0x7fe5e81457f0 2024-03-29T10:55:01.350 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.364+0000 7fe5e4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:01.351 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.364+0000 7fe5e4ff9640 1 -- 172.21.15.112:0/147016070 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe5e0003b10 con 0x7fe5e81457f0 2024-03-29T10:55:01.351 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.365+0000 7fe5e4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:01.351 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.365+0000 7fe5e4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:01.351 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.365+0000 7fe5e4ff9640 10 monclient: dump: 2024-03-29T10:55:01.351 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:01.351 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:01.351 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:01.351 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:01.351 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:01.351 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:01.351 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:01.351 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:01.351 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:01.351 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:01.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.366+0000 7fe5e67fc640 4 set_mon_vals no callback set 2024-03-29T10:55:01.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.367+0000 7fe5ee7a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:01.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.367+0000 7fe5ee7a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:01.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.367+0000 7fe5ee7a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:55:01.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.367+0000 7fe5ee7a4640 1 -- 172.21.15.112:0/147016070 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5c8005080 con 0x7fe5e81457f0 2024-03-29T10:55:01.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.367+0000 7fe5e4ff9640 1 -- 172.21.15.112:0/147016070 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fe5e0019ae0 con 0x7fe5e81457f0 2024-03-29T10:55:01.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.368+0000 7fe5e7fff640 10 monclient: get_auth_request con 0x7fe5d406dda0 auth_method 0 2024-03-29T10:55:01.353 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.368+0000 7fe5e4ff9640 1 -- 172.21.15.112:0/147016070 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7fe5e0083ad0 con 0x7fe5e81457f0 2024-03-29T10:55:01.354 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.368+0000 7fe5e4ff9640 10 client.4353.objecter ms_dispatch 0x7fe5e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:01.354 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.368+0000 7fe5e4ff9640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:01.354 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.368+0000 7fe5e4ff9640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:01.354 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.368+0000 7fe5e4ff9640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-29T10:55:01.354 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.368+0000 7fe5e4ff9640 10 client.4353.objecter ms_handle_connect 0x7fe5d406dda0 2024-03-29T10:55:01.356 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.375+0000 7fe5e4ff9640 1 -- 172.21.15.112:0/147016070 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe5e0085e60 con 0x7fe5e81457f0 2024-03-29T10:55:01.356 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.375+0000 7fe5e4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:01.356 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.375+0000 7fe5e4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:01.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.698+0000 7fe5ee7a4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:01.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.698+0000 7fe5ee7a4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:01.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.698+0000 7fe5ee7a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:55:01.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.698+0000 7fe5ee7a4640 1 -- 172.21.15.112:0/147016070 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5c8005330 con 0x7fe5e81457f0 2024-03-29T10:55:01.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.700+0000 7fe5e4ff9640 1 -- 172.21.15.112:0/147016070 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7fe5e004f1e0 con 0x7fe5e81457f0 2024-03-29T10:55:01.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.700+0000 7fe5e4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:01.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.700+0000 7fe5e4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:01.681 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:55:01.681 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:01.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.703+0000 7fe5ee7a4640 10 librados: watch_flush enter 2024-03-29T10:55:01.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.703+0000 7fe5ee7a4640 10 librados: watch_flush exit 2024-03-29T10:55:01.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.703+0000 7fe5ee7a4640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-29T10:55:01.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.703+0000 7fe5ee7a4640 10 client.4353.objecter successfully canceled tick 2024-03-29T10:55:01.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.703+0000 7fe5ee7a4640 1 -- 172.21.15.112:0/147016070 >> v1:172.21.15.118:6816/3181073408 conn(0x7fe5d406dda0 legacy=0x7fe5d4070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:01.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.704+0000 7fe5ee7a4640 10 monclient: shutdown 2024-03-29T10:55:01.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.704+0000 7fe5ee7a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:01.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.704+0000 7fe5ee7a4640 1 -- 172.21.15.112:0/147016070 >> v1:172.21.15.112:6789/0 conn(0x7fe5e81457f0 legacy=0x7fe5e8137310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:01.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.704+0000 7fe5ee7a4640 1 -- 172.21.15.112:0/147016070 shutdown_connections 2024-03-29T10:55:01.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.704+0000 7fe5ee7a4640 1 -- 172.21.15.112:0/147016070 >> 172.21.15.112:0/147016070 conn(0x7fe5e8066a00 msgr2=0x7fe5e80b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:01.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.704+0000 7fe5ee7a4640 1 -- 172.21.15.112:0/147016070 shutdown_connections 2024-03-29T10:55:01.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.704+0000 7fe5ee7a4640 1 -- 172.21.15.112:0/147016070 wait complete. 2024-03-29T10:55:01.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:01.704+0000 7fe5ee7a4640 1 librados: shutdown 2024-03-29T10:55:01.706 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:08.513 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:08.670 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.688+0000 7f7c1d88d640 10 monclient: get_monmap_and_config 2024-03-29T10:55:08.671 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.688+0000 7f7c1d88d640 10 monclient: build_initial_monmap 2024-03-29T10:55:08.671 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.688+0000 7f7c1d88d640 10 monclient: monmap: 2024-03-29T10:55:08.671 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:55:08.671 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:08.671 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:55:08.689436+0000 2024-03-29T10:55:08.671 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:55:08.689436+0000 2024-03-29T10:55:08.671 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:08.671 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:08.671 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:55:08.671 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:55:08.671 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:55:08.671 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:08.671 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.689+0000 7f7c1d88d640 1 Processor -- start 2024-03-29T10:55:08.671 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.689+0000 7f7c1d88d640 1 -- start start 2024-03-29T10:55:08.672 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.689+0000 7f7c1d88d640 10 monclient: init 2024-03-29T10:55:08.672 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.690+0000 7f7c1d88d640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:08.672 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.690+0000 7f7c1d88d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:55:08.672 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.690+0000 7f7c1d88d640 10 monclient(hunting): picked mon.noname-b con 0x7f7c180ad6c0 addr v1:172.21.15.112:6790/0 2024-03-29T10:55:08.672 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.690+0000 7f7c1d88d640 10 monclient(hunting): picked mon.noname-a con 0x7f7c18152980 addr v1:172.21.15.112:6789/0 2024-03-29T10:55:08.672 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.690+0000 7f7c1d88d640 10 monclient(hunting): picked mon.noname-c con 0x7f7c181588d0 addr v1:172.21.15.118:6789/0 2024-03-29T10:55:08.672 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.690+0000 7f7c1d88d640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c180a94e0 con 0x7f7c18152980 2024-03-29T10:55:08.672 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.690+0000 7f7c1d88d640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c180af3a0 con 0x7f7c181588d0 2024-03-29T10:55:08.673 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.690+0000 7f7c1d88d640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c180afb60 con 0x7f7c180ad6c0 2024-03-29T10:55:08.673 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.690+0000 7f7c1d88d640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:08.673 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.690+0000 7f7c1d88d640 10 monclient(hunting): authenticate will time out at 1063.994832s 2024-03-29T10:55:08.673 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.691+0000 7f7c16ffd640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f7c180ad6c0 0x7f7c1814fb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:55690/0 (socket says 172.21.15.112:55690) 2024-03-29T10:55:08.673 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.691+0000 7f7c16ffd640 1 -- 172.21.15.112:0/931660398 learned_addr learned my addr 172.21.15.112:0/931660398 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:55:08.673 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.693+0000 7f7c15ffb640 1 -- 172.21.15.112:0/931660398 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500207512 0 0) 0x7f7c180afb60 con 0x7f7c180ad6c0 2024-03-29T10:55:08.673 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.693+0000 7f7c15ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:55:08.674 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.693+0000 7f7c15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:08.674 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.693+0000 7f7c15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:08.674 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.693+0000 7f7c15ffb640 10 monclient(hunting): my global_id is 4370 2024-03-29T10:55:08.674 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.693+0000 7f7c15ffb640 1 -- 172.21.15.112:0/931660398 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c00003520 con 0x7f7c180ad6c0 2024-03-29T10:55:08.675 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.694+0000 7f7c15ffb640 1 -- 172.21.15.112:0/931660398 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1719231463 0 0) 0x7f7c00003520 con 0x7f7c180ad6c0 2024-03-29T10:55:08.675 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.694+0000 7f7c15ffb640 1 -- 172.21.15.112:0/931660398 >> v1:172.21.15.118:6789/0 conn(0x7f7c181588d0 legacy=0x7f7c1815acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:08.678 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.694+0000 7f7c15ffb640 1 -- 172.21.15.112:0/931660398 >> v1:172.21.15.112:6789/0 conn(0x7f7c18152980 legacy=0x7f7c18152df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:08.678 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.694+0000 7f7c15ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:55:08.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.694+0000 7f7c15ffb640 1 monclient: found mon.noname-b 2024-03-29T10:55:08.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.694+0000 7f7c15ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:08.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.694+0000 7f7c15ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:55:08.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.694+0000 7f7c15ffb640 1 -- 172.21.15.112:0/931660398 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c180aab80 con 0x7f7c180ad6c0 2024-03-29T10:55:08.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.694+0000 7f7c15ffb640 10 monclient: _finish_auth 0 2024-03-29T10:55:08.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.694+0000 7f7c15ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:55:08.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.694+0000 7f7c15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:08.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.694+0000 7f7c15ffb640 1 -- 172.21.15.112:0/931660398 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f7c10001a10 con 0x7f7c180ad6c0 2024-03-29T10:55:08.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.694+0000 7f7c15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:08.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.694+0000 7f7c15ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:08.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.694+0000 7f7c15ffb640 10 monclient: dump: 2024-03-29T10:55:08.679 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:08.679 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:08.680 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:08.680 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:08.680 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:08.680 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:08.680 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:08.680 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:08.680 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:08.680 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:08.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.695+0000 7f7c1d88d640 5 monclient: authenticate success, global_id 4370 2024-03-29T10:55:08.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.695+0000 7f7c1d88d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:08.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.695+0000 7f7c15ffb640 1 -- 172.21.15.112:0/931660398 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c10001b70 con 0x7f7c180ad6c0 2024-03-29T10:55:08.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.695+0000 7f7c15ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:08.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.695+0000 7f7c15ffb640 1 -- 172.21.15.112:0/931660398 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f7c10001f80 con 0x7f7c180ad6c0 2024-03-29T10:55:08.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.695+0000 7f7c15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:08.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.695+0000 7f7c15ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:08.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.695+0000 7f7c15ffb640 10 monclient: dump: 2024-03-29T10:55:08.681 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:08.681 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:08.681 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:08.681 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:08.681 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:08.681 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:08.681 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:08.681 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:08.681 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:08.681 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:08.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.695+0000 7f7c1d88d640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:08.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.695+0000 7f7c1d88d640 4 set_mon_vals no callback set 2024-03-29T10:55:08.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.696+0000 7f7c1d88d640 10 monclient: shutdown 2024-03-29T10:55:08.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.696+0000 7f7c1d88d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:08.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.696+0000 7f7c1d88d640 1 -- 172.21.15.112:0/931660398 >> v1:172.21.15.112:6790/0 conn(0x7f7c180ad6c0 legacy=0x7f7c1814fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:08.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.697+0000 7f7c1d88d640 1 -- 172.21.15.112:0/931660398 shutdown_connections 2024-03-29T10:55:08.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.697+0000 7f7c1d88d640 1 -- 172.21.15.112:0/931660398 >> 172.21.15.112:0/931660398 conn(0x7f7c180b35e0 msgr2=0x7f7c180b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:08.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.699+0000 7f7c1d88d640 1 -- 172.21.15.112:0/931660398 shutdown_connections 2024-03-29T10:55:08.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.699+0000 7f7c1d88d640 1 -- 172.21.15.112:0/931660398 wait complete. 2024-03-29T10:55:08.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.700+0000 7f7c1d88d640 10 monclient: build_initial_monmap 2024-03-29T10:55:08.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.700+0000 7f7c1d88d640 10 monclient: monmap: 2024-03-29T10:55:08.687 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:55:08.687 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:08.687 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:55:08.687 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:55:08.687 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:08.688 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:08.688 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:55:08.688 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:55:08.688 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:55:08.688 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:08.688 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.700+0000 7f7c1d88d640 1 librados: starting msgr at 2024-03-29T10:55:08.688 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.700+0000 7f7c1d88d640 1 librados: starting objecter 2024-03-29T10:55:08.688 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.701+0000 7f7c1d88d640 1 Processor -- start 2024-03-29T10:55:08.688 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.701+0000 7f7c1d88d640 1 -- start start 2024-03-29T10:55:08.688 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.701+0000 7f7c1d88d640 1 librados: setting wanted keys 2024-03-29T10:55:08.688 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.701+0000 7f7c1d88d640 1 librados: calling monclient init 2024-03-29T10:55:08.688 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.701+0000 7f7c1d88d640 10 monclient: init 2024-03-29T10:55:08.688 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.702+0000 7f7c1d88d640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:08.688 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.702+0000 7f7c1d88d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:55:08.688 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.702+0000 7f7c1d88d640 10 monclient(hunting): picked mon.noname-a con 0x7f7c180ad6c0 addr v1:172.21.15.112:6789/0 2024-03-29T10:55:08.688 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.702+0000 7f7c1d88d640 10 monclient(hunting): picked mon.noname-c con 0x7f7c18152980 addr v1:172.21.15.112:6790/0 2024-03-29T10:55:08.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.702+0000 7f7c1d88d640 10 monclient(hunting): picked mon.noname-b con 0x7f7c181588d0 addr v1:172.21.15.118:6789/0 2024-03-29T10:55:08.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.702+0000 7f7c1d88d640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c180723a0 con 0x7f7c180ad6c0 2024-03-29T10:55:08.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.702+0000 7f7c1d88d640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c18071220 con 0x7f7c181588d0 2024-03-29T10:55:08.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.702+0000 7f7c1d88d640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c180725d0 con 0x7f7c18152980 2024-03-29T10:55:08.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.702+0000 7f7c1d88d640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:08.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.702+0000 7f7c1d88d640 10 monclient(hunting): authenticate will time out at 1064.006297s 2024-03-29T10:55:08.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.704+0000 7f7c167fc640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f7c18152980 0x7f7c1806f140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:56782/0 (socket says 172.21.15.112:56782) 2024-03-29T10:55:08.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.704+0000 7f7c167fc640 1 -- 172.21.15.112:0/2972705153 learned_addr learned my addr 172.21.15.112:0/2972705153 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:55:08.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.705+0000 7f7bfffff640 10 client.?.objecter ms_handle_connect 0x7f7c18152980 2024-03-29T10:55:08.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.705+0000 7f7bfffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:08.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.705+0000 7f7bfffff640 1 -- 172.21.15.112:0/2972705153 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1556281266 0 0) 0x7f7c180725d0 con 0x7f7c18152980 2024-03-29T10:55:08.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.705+0000 7f7bfffff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:08.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.705+0000 7f7bfffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:08.690 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.705+0000 7f7bfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:08.690 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.705+0000 7f7bfffff640 10 monclient(hunting): my global_id is 4373 2024-03-29T10:55:08.690 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.705+0000 7f7bfffff640 1 -- 172.21.15.112:0/2972705153 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c04004eb0 con 0x7f7c18152980 2024-03-29T10:55:08.690 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.705+0000 7f7bfffff640 1 -- 172.21.15.112:0/2972705153 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2302948780 0 0) 0x7f7c04004eb0 con 0x7f7c18152980 2024-03-29T10:55:08.690 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.705+0000 7f7bfffff640 1 -- 172.21.15.112:0/2972705153 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c180725d0 con 0x7f7c18152980 2024-03-29T10:55:08.690 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.705+0000 7f7bfffff640 1 -- 172.21.15.112:0/2972705153 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f7c08001940 con 0x7f7c18152980 2024-03-29T10:55:08.690 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.705+0000 7f7bfffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:08.690 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.705+0000 7f7bfffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:08.690 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.705+0000 7f7bfffff640 10 monclient(hunting): dump: 2024-03-29T10:55:08.690 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:08.690 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:08.690 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:08.690 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:08.690 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:08.691 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:08.691 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:08.691 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:08.691 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:08.691 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:08.691 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.705+0000 7f7bfffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:08.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.709+0000 7f7bfffff640 10 client.?.objecter ms_handle_connect 0x7f7c181588d0 2024-03-29T10:55:08.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.709+0000 7f7bfffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:08.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.709+0000 7f7bfffff640 10 client.?.objecter ms_handle_connect 0x7f7c180ad6c0 2024-03-29T10:55:08.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.709+0000 7f7bfffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:08.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.709+0000 7f7bfffff640 1 -- 172.21.15.112:0/2972705153 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1875216824 0 0) 0x7f7c180725d0 con 0x7f7c18152980 2024-03-29T10:55:08.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.709+0000 7f7bfffff640 1 -- 172.21.15.112:0/2972705153 >> v1:172.21.15.118:6789/0 conn(0x7f7c181588d0 legacy=0x7f7c1806f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:08.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.709+0000 7f7bfffff640 1 -- 172.21.15.112:0/2972705153 >> v1:172.21.15.112:6789/0 conn(0x7f7c180ad6c0 legacy=0x7f7c18071c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:08.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.709+0000 7f7bfffff640 10 monclient: _finish_hunting 0 2024-03-29T10:55:08.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.709+0000 7f7bfffff640 1 monclient: found mon.c 2024-03-29T10:55:08.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.709+0000 7f7bfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:08.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.709+0000 7f7bfffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:55:08.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.709+0000 7f7bfffff640 1 -- 172.21.15.112:0/2972705153 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c18157e90 con 0x7f7c18152980 2024-03-29T10:55:08.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.709+0000 7f7bfffff640 10 monclient: _finish_auth 0 2024-03-29T10:55:08.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.709+0000 7f7bfffff640 10 monclient: _check_auth_tickets 2024-03-29T10:55:08.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.709+0000 7f7bfffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:08.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.709+0000 7f7c1d88d640 5 monclient: authenticate success, global_id 4373 2024-03-29T10:55:08.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.709+0000 7f7c1d88d640 10 monclient: _renew_subs 2024-03-29T10:55:08.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.709+0000 7f7c1d88d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:55:08.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.709+0000 7f7c1d88d640 1 -- 172.21.15.112:0/2972705153 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c181580e0 con 0x7f7c18152980 2024-03-29T10:55:08.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.709+0000 7f7c1d88d640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:08.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.709+0000 7f7c1d88d640 10 monclient: _renew_subs 2024-03-29T10:55:08.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.709+0000 7f7c1d88d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:55:08.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.709+0000 7f7c1d88d640 1 -- 172.21.15.112:0/2972705153 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c181f4080 con 0x7f7c18152980 2024-03-29T10:55:08.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.709+0000 7f7c1d88d640 1 librados: init done 2024-03-29T10:55:08.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.710+0000 7f7bfffff640 1 -- 172.21.15.112:0/2972705153 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c08001aa0 con 0x7f7c18152980 2024-03-29T10:55:08.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.710+0000 7f7bfffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:08.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.710+0000 7f7bfffff640 1 -- 172.21.15.112:0/2972705153 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f7c08003a60 con 0x7f7c18152980 2024-03-29T10:55:08.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.710+0000 7f7bfffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:08.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.710+0000 7f7bfffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:08.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.710+0000 7f7bfffff640 10 monclient: dump: 2024-03-29T10:55:08.695 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:08.695 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:08.695 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:08.696 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:08.696 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:08.696 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:08.696 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:08.696 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:08.696 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:08.696 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:08.696 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.711+0000 7f7bfdffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:08.696 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.711+0000 7f7bfdffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:08.696 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.711+0000 7f7bfdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:55:08.696 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.711+0000 7f7bfdffb640 1 -- 172.21.15.112:0/2972705153 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c18075f10 con 0x7f7c18152980 2024-03-29T10:55:08.697 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.712+0000 7f7bfffff640 1 -- 172.21.15.112:0/2972705153 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f7c08003bc0 con 0x7f7c18152980 2024-03-29T10:55:08.697 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.712+0000 7f7c157fa640 4 set_mon_vals no callback set 2024-03-29T10:55:08.697 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.714+0000 7f7c16ffd640 10 monclient: get_auth_request con 0x7f7c0406dc50 auth_method 0 2024-03-29T10:55:08.699 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.717+0000 7f7bfffff640 1 -- 172.21.15.112:0/2972705153 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f7c08084b10 con 0x7f7c18152980 2024-03-29T10:55:08.699 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.717+0000 7f7bfffff640 10 client.4373.objecter ms_dispatch 0x7f7c18000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:08.699 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.717+0000 7f7bfffff640 3 client.4373.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:08.699 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.717+0000 7f7bfffff640 3 client.4373.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:08.699 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.717+0000 7f7bfffff640 20 client.4373.objecter dump_active .. 0 homeless 2024-03-29T10:55:08.699 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.718+0000 7f7bfffff640 10 client.4373.objecter ms_handle_connect 0x7f7c0406dc50 2024-03-29T10:55:08.701 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.720+0000 7f7bfffff640 1 -- 172.21.15.112:0/2972705153 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7c080503f0 con 0x7f7c18152980 2024-03-29T10:55:08.701 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.720+0000 7f7bfffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:08.701 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:08.720+0000 7f7bfffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:09.028 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:09.047+0000 7f7bfdffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:09.029 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:09.048+0000 7f7bfdffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:09.029 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:09.048+0000 7f7bfdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:55:09.029 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:09.048+0000 7f7bfdffb640 1 -- 172.21.15.112:0/2972705153 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c180ac470 con 0x7f7c18152980 2024-03-29T10:55:09.029 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:09.048+0000 7f7bfffff640 1 -- 172.21.15.112:0/2972705153 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f7c08052320 con 0x7f7c18152980 2024-03-29T10:55:09.029 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:09.048+0000 7f7bfffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:09.029 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:09.048+0000 7f7bfffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:09.032 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:55:09.032 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:09.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:09.054+0000 7f7bfdffb640 10 librados: watch_flush enter 2024-03-29T10:55:09.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:09.055+0000 7f7bfdffb640 10 librados: watch_flush exit 2024-03-29T10:55:09.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:09.055+0000 7f7bfdffb640 20 client.4373.objecter shutdown clearing up homeless session... 2024-03-29T10:55:09.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:09.055+0000 7f7bfdffb640 10 client.4373.objecter successfully canceled tick 2024-03-29T10:55:09.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:09.055+0000 7f7bfdffb640 1 -- 172.21.15.112:0/2972705153 >> v1:172.21.15.118:6816/3181073408 conn(0x7f7c0406dc50 legacy=0x7f7c04070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:09.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:09.055+0000 7f7bfdffb640 10 monclient: shutdown 2024-03-29T10:55:09.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:09.055+0000 7f7bfdffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:09.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:09.055+0000 7f7bfdffb640 1 -- 172.21.15.112:0/2972705153 >> v1:172.21.15.112:6790/0 conn(0x7f7c18152980 legacy=0x7f7c1806f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:09.037 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:09.057+0000 7f7bfdffb640 1 -- 172.21.15.112:0/2972705153 shutdown_connections 2024-03-29T10:55:09.038 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:09.057+0000 7f7bfdffb640 1 -- 172.21.15.112:0/2972705153 >> 172.21.15.112:0/2972705153 conn(0x7f7c180b35e0 msgr2=0x7f7c180adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:09.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:09.060+0000 7f7bfdffb640 1 -- 172.21.15.112:0/2972705153 shutdown_connections 2024-03-29T10:55:09.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:09.060+0000 7f7bfdffb640 1 -- 172.21.15.112:0/2972705153 wait complete. 2024-03-29T10:55:09.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:09.060+0000 7f7bfdffb640 1 librados: shutdown 2024-03-29T10:55:09.061 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:15.869 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:16.011 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.030+0000 7f09323b6640 10 monclient: get_monmap_and_config 2024-03-29T10:55:16.011 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.030+0000 7f09323b6640 10 monclient: build_initial_monmap 2024-03-29T10:55:16.011 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.030+0000 7f09323b6640 10 monclient: monmap: 2024-03-29T10:55:16.011 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:55:16.011 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:16.011 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:55:16.031196+0000 2024-03-29T10:55:16.011 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:55:16.031196+0000 2024-03-29T10:55:16.011 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:16.011 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:16.012 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:55:16.012 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:55:16.012 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:55:16.012 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:16.014 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.033+0000 7f09323b6640 1 Processor -- start 2024-03-29T10:55:16.015 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.033+0000 7f09323b6640 1 -- start start 2024-03-29T10:55:16.015 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.033+0000 7f09323b6640 10 monclient: init 2024-03-29T10:55:16.018 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.034+0000 7f09323b6640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:16.018 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.034+0000 7f09323b6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:55:16.018 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.034+0000 7f09323b6640 10 monclient(hunting): picked mon.noname-b con 0x7f092c1457f0 addr v1:172.21.15.112:6790/0 2024-03-29T10:55:16.018 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.034+0000 7f09323b6640 10 monclient(hunting): picked mon.noname-a con 0x7f092c13f650 addr v1:172.21.15.112:6789/0 2024-03-29T10:55:16.019 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.034+0000 7f09323b6640 10 monclient(hunting): picked mon.noname-c con 0x7f092c0a9d70 addr v1:172.21.15.118:6789/0 2024-03-29T10:55:16.019 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.034+0000 7f09323b6640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092c0aaef0 con 0x7f092c13f650 2024-03-29T10:55:16.019 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.034+0000 7f09323b6640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092c145bd0 con 0x7f092c0a9d70 2024-03-29T10:55:16.019 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.034+0000 7f09323b6640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092c145eb0 con 0x7f092c1457f0 2024-03-29T10:55:16.019 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.034+0000 7f09323b6640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:16.019 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.034+0000 7f09323b6640 10 monclient(hunting): authenticate will time out at 1071.339119s 2024-03-29T10:55:16.019 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.037+0000 7f092b7fe640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f092c13f650 0x7f092c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:50979/0 (socket says 172.21.15.112:50979) 2024-03-29T10:55:16.019 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.037+0000 7f092b7fe640 1 -- 172.21.15.112:0/1446811545 learned_addr learned my addr 172.21.15.112:0/1446811545 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:55:16.019 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.037+0000 7f092bfff640 1 --1- 172.21.15.112:0/1446811545 >> v1:172.21.15.112:6790/0 conn(0x7f092c1457f0 0x7f092c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:40434/0 (socket says 172.21.15.112:40434) 2024-03-29T10:55:16.020 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 1 -- 172.21.15.112:0/1446811545 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204020641 0 0) 0x7f092c0aaef0 con 0x7f092c13f650 2024-03-29T10:55:16.020 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:16.020 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:16.021 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:16.021 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 10 monclient(hunting): my global_id is 4362 2024-03-29T10:55:16.021 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 1 -- 172.21.15.112:0/1446811545 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0914003520 con 0x7f092c13f650 2024-03-29T10:55:16.021 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 1 -- 172.21.15.112:0/1446811545 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232083432 0 0) 0x7f092c145eb0 con 0x7f092c1457f0 2024-03-29T10:55:16.021 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:16.021 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:16.021 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:16.021 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 10 monclient(hunting): my global_id is 4376 2024-03-29T10:55:16.021 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 1 -- 172.21.15.112:0/1446811545 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f092c0aaef0 con 0x7f092c1457f0 2024-03-29T10:55:16.021 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 1 -- 172.21.15.112:0/1446811545 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4242183761 0 0) 0x7f0914003520 con 0x7f092c13f650 2024-03-29T10:55:16.021 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 1 -- 172.21.15.112:0/1446811545 >> v1:172.21.15.112:6790/0 conn(0x7f092c1457f0 legacy=0x7f092c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:16.021 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 1 -- 172.21.15.112:0/1446811545 >> v1:172.21.15.118:6789/0 conn(0x7f092c0a9d70 legacy=0x7f092c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:55:16.021 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 10 monclient: _finish_hunting 0 2024-03-29T10:55:16.021 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 1 monclient: found mon.noname-a 2024-03-29T10:55:16.022 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:16.022 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T10:55:16.022 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 1 -- 172.21.15.112:0/1446811545 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092c146340 con 0x7f092c13f650 2024-03-29T10:55:16.022 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 10 monclient: _finish_auth 0 2024-03-29T10:55:16.022 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 10 monclient: _check_auth_tickets 2024-03-29T10:55:16.022 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:16.022 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 1 -- 172.21.15.112:0/1446811545 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f0920001690 con 0x7f092c13f650 2024-03-29T10:55:16.022 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:16.022 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:16.022 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f092affd640 10 monclient: dump: 2024-03-29T10:55:16.022 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:16.022 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:16.022 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:16.022 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:16.023 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:16.023 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:16.023 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:16.023 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:16.023 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:16.023 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:16.023 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f09323b6640 5 monclient: authenticate success, global_id 4362 2024-03-29T10:55:16.023 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.038+0000 7f09323b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:16.023 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.039+0000 7f092affd640 1 -- 172.21.15.112:0/1446811545 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09200017f0 con 0x7f092c13f650 2024-03-29T10:55:16.023 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.039+0000 7f092affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:16.023 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.039+0000 7f092affd640 1 -- 172.21.15.112:0/1446811545 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f0920001c00 con 0x7f092c13f650 2024-03-29T10:55:16.023 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.039+0000 7f09323b6640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:16.023 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.039+0000 7f09323b6640 4 set_mon_vals no callback set 2024-03-29T10:55:16.023 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.039+0000 7f092affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:16.024 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.039+0000 7f092affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:16.024 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.039+0000 7f092affd640 10 monclient: dump: 2024-03-29T10:55:16.024 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:16.024 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:16.024 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:16.024 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:16.024 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:16.024 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:16.024 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:16.024 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:16.024 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:16.024 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:16.024 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.039+0000 7f09323b6640 10 monclient: shutdown 2024-03-29T10:55:16.024 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.039+0000 7f09323b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:16.025 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.039+0000 7f09323b6640 1 -- 172.21.15.112:0/1446811545 >> v1:172.21.15.112:6789/0 conn(0x7f092c13f650 legacy=0x7f092c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:16.025 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.040+0000 7f09323b6640 1 -- 172.21.15.112:0/1446811545 shutdown_connections 2024-03-29T10:55:16.025 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.040+0000 7f09323b6640 1 -- 172.21.15.112:0/1446811545 >> 172.21.15.112:0/1446811545 conn(0x7f092c066a00 msgr2=0x7f092c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:16.027 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.040+0000 7f09323b6640 1 -- 172.21.15.112:0/1446811545 shutdown_connections 2024-03-29T10:55:16.027 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.040+0000 7f09323b6640 1 -- 172.21.15.112:0/1446811545 wait complete. 2024-03-29T10:55:16.027 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.041+0000 7f09323b6640 10 monclient: build_initial_monmap 2024-03-29T10:55:16.027 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.041+0000 7f09323b6640 10 monclient: monmap: 2024-03-29T10:55:16.027 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:55:16.027 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:16.027 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:55:16.027 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:55:16.028 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:16.028 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:16.028 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:55:16.028 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:55:16.028 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:55:16.028 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:16.028 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.041+0000 7f09323b6640 1 librados: starting msgr at 2024-03-29T10:55:16.028 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.041+0000 7f09323b6640 1 librados: starting objecter 2024-03-29T10:55:16.028 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.042+0000 7f09323b6640 1 Processor -- start 2024-03-29T10:55:16.028 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.042+0000 7f09323b6640 1 -- start start 2024-03-29T10:55:16.028 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.042+0000 7f09323b6640 1 librados: setting wanted keys 2024-03-29T10:55:16.028 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.042+0000 7f09323b6640 1 librados: calling monclient init 2024-03-29T10:55:16.028 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.042+0000 7f09323b6640 10 monclient: init 2024-03-29T10:55:16.028 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.043+0000 7f09323b6640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:16.028 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.043+0000 7f09323b6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:55:16.029 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.044+0000 7f09323b6640 10 monclient(hunting): picked mon.noname-b con 0x7f092c0a9d70 addr v1:172.21.15.118:6789/0 2024-03-29T10:55:16.029 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.044+0000 7f09323b6640 10 monclient(hunting): picked mon.noname-a con 0x7f092c13f650 addr v1:172.21.15.112:6789/0 2024-03-29T10:55:16.029 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.044+0000 7f09323b6640 10 monclient(hunting): picked mon.noname-c con 0x7f092c1457f0 addr v1:172.21.15.112:6790/0 2024-03-29T10:55:16.029 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.044+0000 7f09323b6640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092c135d30 con 0x7f092c13f650 2024-03-29T10:55:16.029 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.044+0000 7f09323b6640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092c1e8490 con 0x7f092c0a9d70 2024-03-29T10:55:16.029 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.044+0000 7f09323b6640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092c144540 con 0x7f092c1457f0 2024-03-29T10:55:16.029 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.044+0000 7f09323b6640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:16.029 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.044+0000 7f09323b6640 10 monclient(hunting): authenticate will time out at 1071.348762s 2024-03-29T10:55:16.029 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.045+0000 7f0930953640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f092c1457f0 0x7f092c1e7d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41028/0 (socket says 172.21.15.112:41028) 2024-03-29T10:55:16.029 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.045+0000 7f0930953640 1 -- 172.21.15.112:0/2322690148 learned_addr learned my addr 172.21.15.112:0/2322690148 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:55:16.029 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.045+0000 7f0928ff9640 10 client.?.objecter ms_handle_connect 0x7f092c1457f0 2024-03-29T10:55:16.029 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.045+0000 7f0928ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:16.029 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.045+0000 7f0928ff9640 1 -- 172.21.15.112:0/2322690148 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2589487689 0 0) 0x7f092c144540 con 0x7f092c1457f0 2024-03-29T10:55:16.029 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.045+0000 7f0928ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:55:16.030 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.045+0000 7f0928ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:16.030 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.045+0000 7f0928ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:16.030 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.045+0000 7f0928ff9640 10 monclient(hunting): my global_id is 4379 2024-03-29T10:55:16.030 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.045+0000 7f0928ff9640 1 -- 172.21.15.112:0/2322690148 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0918004eb0 con 0x7f092c1457f0 2024-03-29T10:55:16.030 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.046+0000 7f0928ff9640 1 -- 172.21.15.112:0/2322690148 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3395648044 0 0) 0x7f0918004eb0 con 0x7f092c1457f0 2024-03-29T10:55:16.030 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.046+0000 7f0928ff9640 1 -- 172.21.15.112:0/2322690148 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f092c144540 con 0x7f092c1457f0 2024-03-29T10:55:16.030 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.046+0000 7f0928ff9640 1 -- 172.21.15.112:0/2322690148 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f0924004380 con 0x7f092c1457f0 2024-03-29T10:55:16.030 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.046+0000 7f0928ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:16.030 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.046+0000 7f0928ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:16.030 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.046+0000 7f0928ff9640 10 monclient(hunting): dump: 2024-03-29T10:55:16.030 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:16.030 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:16.030 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:16.030 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:16.031 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:16.031 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:16.031 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:16.031 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:16.031 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:16.031 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:16.031 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.046+0000 7f0928ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:16.031 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.046+0000 7f0928ff9640 1 -- 172.21.15.112:0/2322690148 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3384285000 0 0) 0x7f092c144540 con 0x7f092c1457f0 2024-03-29T10:55:16.031 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.046+0000 7f0928ff9640 1 -- 172.21.15.112:0/2322690148 >> v1:172.21.15.118:6789/0 conn(0x7f092c0a9d70 legacy=0x7f092c1e7670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:16.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f0928ff9640 1 -- 172.21.15.112:0/2322690148 >> v1:172.21.15.112:6789/0 conn(0x7f092c13f650 legacy=0x7f092c135620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:16.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f0928ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:55:16.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f0928ff9640 1 monclient: found mon.c 2024-03-29T10:55:16.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f0928ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:16.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f0928ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:55:16.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f0928ff9640 1 -- 172.21.15.112:0/2322690148 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092c1436c0 con 0x7f092c1457f0 2024-03-29T10:55:16.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f0928ff9640 10 monclient: _finish_auth 0 2024-03-29T10:55:16.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f0928ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:55:16.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f0928ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:16.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f0928ff9640 10 client.?.objecter ms_handle_connect 0x7f092c13f650 2024-03-29T10:55:16.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f0928ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:16.033 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f09323b6640 5 monclient: authenticate success, global_id 4379 2024-03-29T10:55:16.033 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f09323b6640 10 monclient: _renew_subs 2024-03-29T10:55:16.033 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f09323b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:55:16.033 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f0928ff9640 1 -- 172.21.15.112:0/2322690148 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09240044e0 con 0x7f092c1457f0 2024-03-29T10:55:16.033 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f09323b6640 1 -- 172.21.15.112:0/2322690148 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f092c1432e0 con 0x7f092c1457f0 2024-03-29T10:55:16.033 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f0928ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:16.033 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f0928ff9640 1 -- 172.21.15.112:0/2322690148 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f09240014f0 con 0x7f092c1457f0 2024-03-29T10:55:16.033 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f0928ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:16.033 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f0928ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:16.033 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f0928ff9640 10 monclient: dump: 2024-03-29T10:55:16.033 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:16.033 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:16.033 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:16.033 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:16.034 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:16.034 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:16.034 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:16.034 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:16.034 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:16.034 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:16.034 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f09323b6640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:16.034 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f09323b6640 10 monclient: _renew_subs 2024-03-29T10:55:16.034 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f09323b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:55:16.034 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f09323b6640 1 -- 172.21.15.112:0/2322690148 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f092c13e9e0 con 0x7f092c1457f0 2024-03-29T10:55:16.034 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f09323b6640 1 librados: init done 2024-03-29T10:55:16.034 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.049+0000 7f092a7fc640 4 set_mon_vals no callback set 2024-03-29T10:55:16.035 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.051+0000 7f0928ff9640 1 -- 172.21.15.112:0/2322690148 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f0924001650 con 0x7f092c1457f0 2024-03-29T10:55:16.035 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.052+0000 7f0928ff9640 1 -- 172.21.15.112:0/2322690148 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f0924084de0 con 0x7f092c1457f0 2024-03-29T10:55:16.035 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.052+0000 7f0928ff9640 10 client.4379.objecter ms_dispatch 0x7f092c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:16.035 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.052+0000 7f0928ff9640 3 client.4379.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:16.035 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.052+0000 7f0928ff9640 3 client.4379.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:16.035 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.052+0000 7f0928ff9640 20 client.4379.objecter dump_active .. 0 homeless 2024-03-29T10:55:16.035 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.053+0000 7f092bfff640 10 monclient: get_auth_request con 0x7f091806dbb0 auth_method 0 2024-03-29T10:55:16.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.053+0000 7f0928ff9640 10 client.4379.objecter ms_handle_connect 0x7f091806dbb0 2024-03-29T10:55:16.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.055+0000 7f09323b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:16.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.055+0000 7f09323b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:16.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.055+0000 7f09323b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:55:16.036 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.055+0000 7f09323b6640 1 -- 172.21.15.112:0/2322690148 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f090c005080 con 0x7f092c1457f0 2024-03-29T10:55:16.046 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.065+0000 7f0928ff9640 1 -- 172.21.15.112:0/2322690148 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f09240504f0 con 0x7f092c1457f0 2024-03-29T10:55:16.046 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.066+0000 7f0928ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:16.047 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.066+0000 7f0928ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:16.372 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.391+0000 7f09323b6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:16.373 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.391+0000 7f09323b6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:16.373 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.391+0000 7f09323b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:55:16.373 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.391+0000 7f09323b6640 1 -- 172.21.15.112:0/2322690148 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f090c005330 con 0x7f092c1457f0 2024-03-29T10:55:16.374 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.392+0000 7f0928ff9640 1 -- 172.21.15.112:0/2322690148 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f09240525f0 con 0x7f092c1457f0 2024-03-29T10:55:16.374 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.392+0000 7f0928ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:16.374 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.392+0000 7f0928ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:16.374 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:55:16.374 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:16.376 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.395+0000 7f09323b6640 10 librados: watch_flush enter 2024-03-29T10:55:16.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.396+0000 7f09323b6640 10 librados: watch_flush exit 2024-03-29T10:55:16.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.396+0000 7f09323b6640 20 client.4379.objecter shutdown clearing up homeless session... 2024-03-29T10:55:16.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.396+0000 7f09323b6640 10 client.4379.objecter successfully canceled tick 2024-03-29T10:55:16.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.396+0000 7f09323b6640 1 -- 172.21.15.112:0/2322690148 >> v1:172.21.15.118:6816/3181073408 conn(0x7f091806dbb0 legacy=0x7f0918070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:16.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.396+0000 7f09323b6640 10 monclient: shutdown 2024-03-29T10:55:16.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.396+0000 7f09323b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:16.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.396+0000 7f09323b6640 1 -- 172.21.15.112:0/2322690148 >> v1:172.21.15.112:6790/0 conn(0x7f092c1457f0 legacy=0x7f092c1e7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:16.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.396+0000 7f09323b6640 1 -- 172.21.15.112:0/2322690148 shutdown_connections 2024-03-29T10:55:16.377 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.396+0000 7f09323b6640 1 -- 172.21.15.112:0/2322690148 >> 172.21.15.112:0/2322690148 conn(0x7f092c066a00 msgr2=0x7f092c13a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:16.378 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.396+0000 7f09323b6640 1 -- 172.21.15.112:0/2322690148 shutdown_connections 2024-03-29T10:55:16.378 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.396+0000 7f09323b6640 1 -- 172.21.15.112:0/2322690148 wait complete. 2024-03-29T10:55:16.379 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:16.396+0000 7f09323b6640 1 librados: shutdown 2024-03-29T10:55:16.394 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:23.201 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:23.362 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.381+0000 7fd90a3af640 10 monclient: get_monmap_and_config 2024-03-29T10:55:23.362 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.381+0000 7fd90a3af640 10 monclient: build_initial_monmap 2024-03-29T10:55:23.362 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.381+0000 7fd90a3af640 10 monclient: monmap: 2024-03-29T10:55:23.363 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:55:23.363 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:23.363 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:55:23.382621+0000 2024-03-29T10:55:23.363 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:55:23.382621+0000 2024-03-29T10:55:23.363 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:23.363 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:23.363 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:55:23.363 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:55:23.363 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:55:23.363 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:23.367 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.384+0000 7fd90a3af640 1 Processor -- start 2024-03-29T10:55:23.367 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.384+0000 7fd90a3af640 1 -- start start 2024-03-29T10:55:23.367 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.384+0000 7fd90a3af640 10 monclient: init 2024-03-29T10:55:23.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.385+0000 7fd90a3af640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:23.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.385+0000 7fd90a3af640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:55:23.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.385+0000 7fd90a3af640 10 monclient(hunting): picked mon.noname-b con 0x7fd9041457f0 addr v1:172.21.15.112:6790/0 2024-03-29T10:55:23.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.385+0000 7fd90a3af640 10 monclient(hunting): picked mon.noname-c con 0x7fd90413f650 addr v1:172.21.15.118:6789/0 2024-03-29T10:55:23.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.385+0000 7fd90a3af640 10 monclient(hunting): picked mon.noname-a con 0x7fd9040a9d70 addr v1:172.21.15.112:6789/0 2024-03-29T10:55:23.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.385+0000 7fd90a3af640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9040aaef0 con 0x7fd9040a9d70 2024-03-29T10:55:23.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.385+0000 7fd90a3af640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd904145bd0 con 0x7fd90413f650 2024-03-29T10:55:23.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.385+0000 7fd90a3af640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd904145eb0 con 0x7fd9041457f0 2024-03-29T10:55:23.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.385+0000 7fd90a3af640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:23.368 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.385+0000 7fd90a3af640 10 monclient(hunting): authenticate will time out at 1078.689483s 2024-03-29T10:55:23.370 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.388+0000 7fd903fff640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fd9041457f0 0x7fd90413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:39123/0 (socket says 172.21.15.112:39123) 2024-03-29T10:55:23.370 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.389+0000 7fd903fff640 1 -- 172.21.15.112:0/3064328993 learned_addr learned my addr 172.21.15.112:0/3064328993 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:55:23.371 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.390+0000 7fd902ffd640 1 -- 172.21.15.112:0/3064328993 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252990849 0 0) 0x7fd904145bd0 con 0x7fd90413f650 2024-03-29T10:55:23.371 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.390+0000 7fd902ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:23.371 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.390+0000 7fd902ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:23.371 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.390+0000 7fd902ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:23.371 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.390+0000 7fd902ffd640 10 monclient(hunting): my global_id is 4354 2024-03-29T10:55:23.371 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.390+0000 7fd902ffd640 1 -- 172.21.15.112:0/3064328993 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8ec003520 con 0x7fd90413f650 2024-03-29T10:55:23.374 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.393+0000 7fd902ffd640 1 -- 172.21.15.112:0/3064328993 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723999156 0 0) 0x7fd904145eb0 con 0x7fd9041457f0 2024-03-29T10:55:23.374 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.393+0000 7fd902ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:23.374 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.393+0000 7fd902ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:23.374 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.393+0000 7fd902ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:23.374 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.393+0000 7fd902ffd640 10 monclient(hunting): my global_id is 4382 2024-03-29T10:55:23.374 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.393+0000 7fd902ffd640 1 -- 172.21.15.112:0/3064328993 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd904145bd0 con 0x7fd9041457f0 2024-03-29T10:55:23.374 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.393+0000 7fd902ffd640 1 -- 172.21.15.112:0/3064328993 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 882456124 0 0) 0x7fd8ec003520 con 0x7fd90413f650 2024-03-29T10:55:23.374 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.393+0000 7fd902ffd640 1 -- 172.21.15.112:0/3064328993 >> v1:172.21.15.112:6790/0 conn(0x7fd9041457f0 legacy=0x7fd90413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:23.374 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.393+0000 7fd902ffd640 1 -- 172.21.15.112:0/3064328993 >> v1:172.21.15.112:6789/0 conn(0x7fd9040a9d70 legacy=0x7fd90414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:23.378 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.396+0000 7fd902ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:55:23.378 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.396+0000 7fd902ffd640 1 monclient: found mon.noname-c 2024-03-29T10:55:23.378 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.396+0000 7fd902ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:23.378 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.396+0000 7fd902ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T10:55:23.378 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.396+0000 7fd902ffd640 1 -- 172.21.15.112:0/3064328993 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd904146340 con 0x7fd90413f650 2024-03-29T10:55:23.378 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.396+0000 7fd902ffd640 10 monclient: _finish_auth 0 2024-03-29T10:55:23.378 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.396+0000 7fd902ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:55:23.378 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.396+0000 7fd902ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:23.378 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.396+0000 7fd902ffd640 1 -- 172.21.15.112:0/3064328993 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fd8fc001990 con 0x7fd90413f650 2024-03-29T10:55:23.378 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.396+0000 7fd902ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:23.378 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.396+0000 7fd902ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:23.378 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.396+0000 7fd902ffd640 10 monclient: dump: 2024-03-29T10:55:23.379 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:23.379 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:23.379 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:23.379 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:23.379 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:23.379 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:23.379 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:23.379 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:23.379 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:23.379 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:23.379 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.396+0000 7fd90a3af640 5 monclient: authenticate success, global_id 4354 2024-03-29T10:55:23.379 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.396+0000 7fd90a3af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:23.381 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.397+0000 7fd902ffd640 1 -- 172.21.15.112:0/3064328993 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8fc001af0 con 0x7fd90413f650 2024-03-29T10:55:23.381 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.397+0000 7fd902ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:23.381 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.397+0000 7fd90a3af640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:23.381 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.397+0000 7fd90a3af640 4 set_mon_vals no callback set 2024-03-29T10:55:23.381 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.397+0000 7fd902ffd640 1 -- 172.21.15.112:0/3064328993 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fd8fc001f00 con 0x7fd90413f650 2024-03-29T10:55:23.381 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.397+0000 7fd902ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:23.381 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.397+0000 7fd902ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:23.381 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.397+0000 7fd902ffd640 10 monclient: dump: 2024-03-29T10:55:23.381 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:23.381 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:23.381 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:23.382 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:23.382 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:23.382 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:23.382 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:23.382 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:23.382 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:23.382 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:23.382 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.398+0000 7fd90a3af640 10 monclient: shutdown 2024-03-29T10:55:23.382 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.398+0000 7fd90a3af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:23.382 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.398+0000 7fd90a3af640 1 -- 172.21.15.112:0/3064328993 >> v1:172.21.15.118:6789/0 conn(0x7fd90413f650 legacy=0x7fd9040b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:23.382 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.398+0000 7fd90a3af640 1 -- 172.21.15.112:0/3064328993 shutdown_connections 2024-03-29T10:55:23.382 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.398+0000 7fd90a3af640 1 -- 172.21.15.112:0/3064328993 >> 172.21.15.112:0/3064328993 conn(0x7fd904066a00 msgr2=0x7fd904066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:23.383 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.402+0000 7fd90a3af640 1 -- 172.21.15.112:0/3064328993 shutdown_connections 2024-03-29T10:55:23.383 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.402+0000 7fd90a3af640 1 -- 172.21.15.112:0/3064328993 wait complete. 2024-03-29T10:55:23.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.403+0000 7fd90a3af640 10 monclient: build_initial_monmap 2024-03-29T10:55:23.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.403+0000 7fd90a3af640 10 monclient: monmap: 2024-03-29T10:55:23.386 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:55:23.387 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:23.387 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:55:23.387 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:55:23.387 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:23.387 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:23.387 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:55:23.387 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:55:23.387 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:55:23.387 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:23.387 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.403+0000 7fd90a3af640 1 librados: starting msgr at 2024-03-29T10:55:23.387 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.404+0000 7fd90a3af640 1 librados: starting objecter 2024-03-29T10:55:23.387 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.404+0000 7fd90a3af640 1 Processor -- start 2024-03-29T10:55:23.387 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.404+0000 7fd90a3af640 1 -- start start 2024-03-29T10:55:23.387 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.404+0000 7fd90a3af640 1 librados: setting wanted keys 2024-03-29T10:55:23.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.404+0000 7fd90a3af640 1 librados: calling monclient init 2024-03-29T10:55:23.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.404+0000 7fd90a3af640 10 monclient: init 2024-03-29T10:55:23.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.405+0000 7fd90a3af640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:23.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.405+0000 7fd90a3af640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:55:23.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.405+0000 7fd90a3af640 10 monclient(hunting): picked mon.noname-c con 0x7fd9040a9d70 addr v1:172.21.15.112:6790/0 2024-03-29T10:55:23.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.405+0000 7fd90a3af640 10 monclient(hunting): picked mon.noname-b con 0x7fd90413f650 addr v1:172.21.15.118:6789/0 2024-03-29T10:55:23.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.405+0000 7fd90a3af640 10 monclient(hunting): picked mon.noname-a con 0x7fd9041457f0 addr v1:172.21.15.112:6789/0 2024-03-29T10:55:23.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.405+0000 7fd90a3af640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd904140ce0 con 0x7fd9041457f0 2024-03-29T10:55:23.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.405+0000 7fd90a3af640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd904141e40 con 0x7fd90413f650 2024-03-29T10:55:23.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.405+0000 7fd90a3af640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9041356e0 con 0x7fd9040a9d70 2024-03-29T10:55:23.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.405+0000 7fd90a3af640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:23.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.405+0000 7fd90a3af640 10 monclient(hunting): authenticate will time out at 1078.709576s 2024-03-29T10:55:23.395 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.414+0000 7fd9037fe640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fd90413f650 0x7fd904134fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:43955/0 (socket says 172.21.15.112:43955) 2024-03-29T10:55:23.395 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.414+0000 7fd9037fe640 1 -- 172.21.15.112:0/1146143780 learned_addr learned my addr 172.21.15.112:0/1146143780 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:55:23.396 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.414+0000 7fd900ff9640 10 client.?.objecter ms_handle_connect 0x7fd90413f650 2024-03-29T10:55:23.396 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.414+0000 7fd900ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:23.396 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.415+0000 7fd900ff9640 1 -- 172.21.15.112:0/1146143780 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871411119 0 0) 0x7fd904141e40 con 0x7fd90413f650 2024-03-29T10:55:23.396 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.415+0000 7fd900ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:55:23.396 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.415+0000 7fd900ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:23.396 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.415+0000 7fd900ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:23.396 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.415+0000 7fd900ff9640 10 monclient(hunting): my global_id is 4357 2024-03-29T10:55:23.396 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.415+0000 7fd900ff9640 1 -- 172.21.15.112:0/1146143780 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8f0004eb0 con 0x7fd90413f650 2024-03-29T10:55:23.396 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.415+0000 7fd900ff9640 1 -- 172.21.15.112:0/1146143780 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1482312215 0 0) 0x7fd8f0004eb0 con 0x7fd90413f650 2024-03-29T10:55:23.397 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.415+0000 7fd900ff9640 1 -- 172.21.15.112:0/1146143780 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd904141e40 con 0x7fd90413f650 2024-03-29T10:55:23.397 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.415+0000 7fd900ff9640 1 -- 172.21.15.112:0/1146143780 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fd8fc001f00 con 0x7fd90413f650 2024-03-29T10:55:23.397 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.415+0000 7fd900ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:23.397 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.416+0000 7fd900ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:23.397 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.416+0000 7fd900ff9640 10 monclient(hunting): dump: 2024-03-29T10:55:23.397 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:23.397 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:23.397 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:23.397 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:23.398 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:23.398 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:23.398 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:23.398 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:23.398 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:23.398 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:23.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.416+0000 7fd900ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:23.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.416+0000 7fd900ff9640 1 -- 172.21.15.112:0/1146143780 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 322788512 0 0) 0x7fd904141e40 con 0x7fd90413f650 2024-03-29T10:55:23.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.416+0000 7fd900ff9640 1 -- 172.21.15.112:0/1146143780 >> v1:172.21.15.112:6790/0 conn(0x7fd9040a9d70 legacy=0x7fd904142260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:23.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.416+0000 7fd900ff9640 1 -- 172.21.15.112:0/1146143780 >> v1:172.21.15.112:6789/0 conn(0x7fd9041457f0 legacy=0x7fd90413fe30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:23.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.416+0000 7fd900ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:55:23.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.416+0000 7fd900ff9640 1 monclient: found mon.b 2024-03-29T10:55:23.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.416+0000 7fd900ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:23.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.416+0000 7fd900ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:55:23.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.416+0000 7fd900ff9640 1 -- 172.21.15.112:0/1146143780 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd904135870 con 0x7fd90413f650 2024-03-29T10:55:23.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.416+0000 7fd900ff9640 10 monclient: _finish_auth 0 2024-03-29T10:55:23.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.416+0000 7fd900ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:55:23.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.416+0000 7fd900ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:23.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.417+0000 7fd900ff9640 1 -- 172.21.15.112:0/1146143780 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8fc003d60 con 0x7fd90413f650 2024-03-29T10:55:23.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.417+0000 7fd900ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:23.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.417+0000 7fd900ff9640 1 -- 172.21.15.112:0/1146143780 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fd8fc0041b0 con 0x7fd90413f650 2024-03-29T10:55:23.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.417+0000 7fd900ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:23.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.417+0000 7fd900ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:23.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.417+0000 7fd900ff9640 10 monclient: dump: 2024-03-29T10:55:23.400 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:23.400 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:23.400 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:23.400 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:23.400 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:23.400 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:23.400 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:23.400 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:23.401 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:23.401 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:23.401 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.420+0000 7fd90a3af640 5 monclient: authenticate success, global_id 4357 2024-03-29T10:55:23.401 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.420+0000 7fd90a3af640 10 monclient: _renew_subs 2024-03-29T10:55:23.401 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.420+0000 7fd90a3af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:55:23.401 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.420+0000 7fd90a3af640 1 -- 172.21.15.112:0/1146143780 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90413e030 con 0x7fd90413f650 2024-03-29T10:55:23.401 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.420+0000 7fd90a3af640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:23.401 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.420+0000 7fd90a3af640 10 monclient: _renew_subs 2024-03-29T10:55:23.402 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.420+0000 7fd90a3af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:55:23.402 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.420+0000 7fd90a3af640 1 -- 172.21.15.112:0/1146143780 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd904140540 con 0x7fd90413f650 2024-03-29T10:55:23.402 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.420+0000 7fd90a3af640 1 librados: init done 2024-03-29T10:55:23.402 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.420+0000 7fd901ffb640 4 set_mon_vals no callback set 2024-03-29T10:55:23.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.422+0000 7fd900ff9640 1 -- 172.21.15.112:0/1146143780 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fd8fc004310 con 0x7fd90413f650 2024-03-29T10:55:23.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.422+0000 7fd90a3af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:23.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.422+0000 7fd90a3af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:23.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.422+0000 7fd90a3af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:55:23.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.422+0000 7fd90a3af640 1 -- 172.21.15.112:0/1146143780 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e4005080 con 0x7fd90413f650 2024-03-29T10:55:23.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.423+0000 7fd900ff9640 1 -- 172.21.15.112:0/1146143780 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7fd8fc085000 con 0x7fd90413f650 2024-03-29T10:55:23.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.423+0000 7fd900ff9640 10 client.4357.objecter ms_dispatch 0x7fd904000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:23.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.423+0000 7fd900ff9640 3 client.4357.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:23.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.423+0000 7fd900ff9640 3 client.4357.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:23.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.423+0000 7fd900ff9640 20 client.4357.objecter dump_active .. 0 homeless 2024-03-29T10:55:23.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.424+0000 7fd903fff640 10 monclient: get_auth_request con 0x7fd8f006dc90 auth_method 0 2024-03-29T10:55:23.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.426+0000 7fd900ff9640 10 client.4357.objecter ms_handle_connect 0x7fd8f006dc90 2024-03-29T10:55:23.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.430+0000 7fd900ff9640 1 -- 172.21.15.112:0/1146143780 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd8fc0508e0 con 0x7fd90413f650 2024-03-29T10:55:23.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.430+0000 7fd900ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:23.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.430+0000 7fd900ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:23.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.778+0000 7fd90a3af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:23.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.778+0000 7fd90a3af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:23.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.778+0000 7fd90a3af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:55:23.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.778+0000 7fd90a3af640 1 -- 172.21.15.112:0/1146143780 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8e4005330 con 0x7fd90413f650 2024-03-29T10:55:23.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.779+0000 7fd900ff9640 1 -- 172.21.15.112:0/1146143780 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7fd8fc052810 con 0x7fd90413f650 2024-03-29T10:55:23.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.779+0000 7fd900ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:23.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.779+0000 7fd900ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:23.763 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:55:23.763 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:23.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.786+0000 7fd90a3af640 10 librados: watch_flush enter 2024-03-29T10:55:23.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.786+0000 7fd90a3af640 10 librados: watch_flush exit 2024-03-29T10:55:23.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.786+0000 7fd90a3af640 20 client.4357.objecter shutdown clearing up homeless session... 2024-03-29T10:55:23.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.786+0000 7fd90a3af640 10 client.4357.objecter successfully canceled tick 2024-03-29T10:55:23.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.786+0000 7fd90a3af640 1 -- 172.21.15.112:0/1146143780 >> v1:172.21.15.118:6816/3181073408 conn(0x7fd8f006dc90 legacy=0x7fd8f0070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:23.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.786+0000 7fd90a3af640 10 monclient: shutdown 2024-03-29T10:55:23.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.786+0000 7fd90a3af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:23.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.786+0000 7fd90a3af640 1 -- 172.21.15.112:0/1146143780 >> v1:172.21.15.118:6789/0 conn(0x7fd90413f650 legacy=0x7fd904134fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:23.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.788+0000 7fd90a3af640 1 -- 172.21.15.112:0/1146143780 shutdown_connections 2024-03-29T10:55:23.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.788+0000 7fd90a3af640 1 -- 172.21.15.112:0/1146143780 >> 172.21.15.112:0/1146143780 conn(0x7fd904066a00 msgr2=0x7fd9040b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:23.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.788+0000 7fd90a3af640 1 -- 172.21.15.112:0/1146143780 shutdown_connections 2024-03-29T10:55:23.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.788+0000 7fd90a3af640 1 -- 172.21.15.112:0/1146143780 wait complete. 2024-03-29T10:55:23.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:23.788+0000 7fd90a3af640 1 librados: shutdown 2024-03-29T10:55:23.794 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:30.601 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:30.747 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.766+0000 7f8fed001640 10 monclient: get_monmap_and_config 2024-03-29T10:55:30.747 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.766+0000 7f8fed001640 10 monclient: build_initial_monmap 2024-03-29T10:55:30.748 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.766+0000 7f8fed001640 10 monclient: monmap: 2024-03-29T10:55:30.748 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:55:30.748 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:30.748 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:55:30.767825+0000 2024-03-29T10:55:30.748 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:55:30.767825+0000 2024-03-29T10:55:30.748 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:30.748 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:30.748 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:55:30.748 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:55:30.748 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:55:30.748 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:30.749 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.769+0000 7f8fed001640 1 Processor -- start 2024-03-29T10:55:30.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.769+0000 7f8fed001640 1 -- start start 2024-03-29T10:55:30.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.769+0000 7f8fed001640 10 monclient: init 2024-03-29T10:55:30.752 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.772+0000 7f8fed001640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:30.752 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.772+0000 7f8fed001640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:55:30.752 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.772+0000 7f8fed001640 10 monclient(hunting): picked mon.noname-b con 0x7f8fe8149b50 addr v1:172.21.15.112:6790/0 2024-03-29T10:55:30.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.772+0000 7f8fed001640 10 monclient(hunting): picked mon.noname-a con 0x7f8fe80b2620 addr v1:172.21.15.112:6789/0 2024-03-29T10:55:30.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.772+0000 7f8fed001640 10 monclient(hunting): picked mon.noname-c con 0x7f8fe814fc70 addr v1:172.21.15.118:6789/0 2024-03-29T10:55:30.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.772+0000 7f8fed001640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe806eee0 con 0x7f8fe80b2620 2024-03-29T10:55:30.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.772+0000 7f8fed001640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe806f1c0 con 0x7f8fe814fc70 2024-03-29T10:55:30.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.772+0000 7f8fed001640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe8071d80 con 0x7f8fe8149b50 2024-03-29T10:55:30.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.772+0000 7f8fed001640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:30.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.772+0000 7f8fed001640 10 monclient(hunting): authenticate will time out at 1086.076303s 2024-03-29T10:55:30.756 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.775+0000 7f8fe659c640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f8fe80b2620 0x7f8fe80b2a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:45096/0 (socket says 172.21.15.112:45096) 2024-03-29T10:55:30.756 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.775+0000 7f8fe659c640 1 -- 172.21.15.112:0/1453760045 learned_addr learned my addr 172.21.15.112:0/1453760045 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:55:30.757 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.776+0000 7f8fe5d9b640 1 -- 172.21.15.112:0/1453760045 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180103433 0 0) 0x7f8fe806eee0 con 0x7f8fe80b2620 2024-03-29T10:55:30.757 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.776+0000 7f8fe5d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:55:30.757 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.776+0000 7f8fe5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:30.757 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.776+0000 7f8fe5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:30.757 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.776+0000 7f8fe5d9b640 10 monclient(hunting): my global_id is 4368 2024-03-29T10:55:30.757 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.776+0000 7f8fe5d9b640 1 -- 172.21.15.112:0/1453760045 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd0003520 con 0x7f8fe80b2620 2024-03-29T10:55:30.758 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.777+0000 7f8fe5d9b640 1 -- 172.21.15.112:0/1453760045 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1469940364 0 0) 0x7f8fd0003520 con 0x7f8fe80b2620 2024-03-29T10:55:30.758 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.777+0000 7f8fe5d9b640 1 -- 172.21.15.112:0/1453760045 >> v1:172.21.15.112:6790/0 conn(0x7f8fe8149b50 legacy=0x7f8fe8143440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:30.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.781+0000 7f8fe5d9b640 1 -- 172.21.15.112:0/1453760045 >> v1:172.21.15.118:6789/0 conn(0x7f8fe814fc70 legacy=0x7f8fe8152060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:30.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.781+0000 7f8fe5d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:55:30.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.781+0000 7f8fe5d9b640 1 monclient: found mon.noname-a 2024-03-29T10:55:30.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.781+0000 7f8fe5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:30.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.781+0000 7f8fe5d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T10:55:30.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.781+0000 7f8fe5d9b640 1 -- 172.21.15.112:0/1453760045 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe814a590 con 0x7f8fe80b2620 2024-03-29T10:55:30.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.781+0000 7f8fe5d9b640 10 monclient: _finish_auth 0 2024-03-29T10:55:30.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.781+0000 7f8fe5d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:55:30.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.781+0000 7f8fe5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:30.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.781+0000 7f8fe5d9b640 1 -- 172.21.15.112:0/1453760045 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f8fdc001690 con 0x7f8fe80b2620 2024-03-29T10:55:30.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.781+0000 7f8fe5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:30.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.781+0000 7f8fe5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:30.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.781+0000 7f8fe5d9b640 10 monclient: dump: 2024-03-29T10:55:30.763 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:30.763 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:30.763 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:30.763 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:30.763 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:30.763 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:30.763 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:30.764 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:30.764 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:30.764 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:30.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.781+0000 7f8fed001640 5 monclient: authenticate success, global_id 4368 2024-03-29T10:55:30.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.781+0000 7f8fed001640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:30.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.781+0000 7f8fe5d9b640 1 -- 172.21.15.112:0/1453760045 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fdc001860 con 0x7f8fe80b2620 2024-03-29T10:55:30.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.781+0000 7f8fe5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:30.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.781+0000 7f8fed001640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:30.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.781+0000 7f8fed001640 4 set_mon_vals no callback set 2024-03-29T10:55:30.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.782+0000 7f8fe5d9b640 1 -- 172.21.15.112:0/1453760045 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f8fdc001c70 con 0x7f8fe80b2620 2024-03-29T10:55:30.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.782+0000 7f8fed001640 10 monclient: shutdown 2024-03-29T10:55:30.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.782+0000 7f8fed001640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:30.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.782+0000 7f8fed001640 1 -- 172.21.15.112:0/1453760045 >> v1:172.21.15.112:6789/0 conn(0x7f8fe80b2620 legacy=0x7f8fe80b2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:30.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.782+0000 7f8fe5d9b640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:55:30.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.783+0000 7f8fed001640 1 -- 172.21.15.112:0/1453760045 shutdown_connections 2024-03-29T10:55:30.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.783+0000 7f8fed001640 1 -- 172.21.15.112:0/1453760045 >> 172.21.15.112:0/1453760045 conn(0x7f8fe806d180 msgr2=0x7f8fe8066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:30.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.784+0000 7f8fed001640 1 -- 172.21.15.112:0/1453760045 shutdown_connections 2024-03-29T10:55:30.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.784+0000 7f8fed001640 1 -- 172.21.15.112:0/1453760045 wait complete. 2024-03-29T10:55:30.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.785+0000 7f8fed001640 10 monclient: build_initial_monmap 2024-03-29T10:55:30.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.785+0000 7f8fed001640 10 monclient: monmap: 2024-03-29T10:55:30.766 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:55:30.766 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:30.766 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:55:30.766 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:55:30.766 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:30.767 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:30.767 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:55:30.767 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:55:30.767 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:55:30.767 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:30.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.785+0000 7f8fed001640 1 librados: starting msgr at 2024-03-29T10:55:30.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.785+0000 7f8fed001640 1 librados: starting objecter 2024-03-29T10:55:30.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.786+0000 7f8fed001640 1 Processor -- start 2024-03-29T10:55:30.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.786+0000 7f8fed001640 1 -- start start 2024-03-29T10:55:30.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.786+0000 7f8fed001640 1 librados: setting wanted keys 2024-03-29T10:55:30.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.786+0000 7f8fed001640 1 librados: calling monclient init 2024-03-29T10:55:30.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.786+0000 7f8fed001640 10 monclient: init 2024-03-29T10:55:30.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.787+0000 7f8fed001640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:30.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.787+0000 7f8fed001640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:55:30.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.787+0000 7f8fed001640 10 monclient(hunting): picked mon.noname-a con 0x7f8fe80b2620 addr v1:172.21.15.112:6789/0 2024-03-29T10:55:30.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.787+0000 7f8fed001640 10 monclient(hunting): picked mon.noname-c con 0x7f8fe8149b50 addr v1:172.21.15.112:6790/0 2024-03-29T10:55:30.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.787+0000 7f8fed001640 10 monclient(hunting): picked mon.noname-b con 0x7f8fe814fc70 addr v1:172.21.15.118:6789/0 2024-03-29T10:55:30.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.787+0000 7f8fed001640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe813b840 con 0x7f8fe80b2620 2024-03-29T10:55:30.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.787+0000 7f8fed001640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe813ee30 con 0x7f8fe814fc70 2024-03-29T10:55:30.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.787+0000 7f8fed001640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe806fef0 con 0x7f8fe8149b50 2024-03-29T10:55:30.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.787+0000 7f8fed001640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:30.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.787+0000 7f8fed001640 10 monclient(hunting): authenticate will time out at 1086.091348s 2024-03-29T10:55:30.770 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.789+0000 7f8fe6d9d640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f8fe80b2620 0x7f8fe813f350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:45410/0 (socket says 172.21.15.112:45410) 2024-03-29T10:55:30.770 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.789+0000 7f8fe6d9d640 1 -- 172.21.15.112:0/1327180317 learned_addr learned my addr 172.21.15.112:0/1327180317 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:55:30.770 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.789+0000 7f8fcf7fe640 10 client.?.objecter ms_handle_connect 0x7f8fe80b2620 2024-03-29T10:55:30.770 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.789+0000 7f8fcf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:30.771 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.790+0000 7f8fcf7fe640 1 -- 172.21.15.112:0/1327180317 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352306751 0 0) 0x7f8fe813b840 con 0x7f8fe80b2620 2024-03-29T10:55:30.771 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.790+0000 7f8fcf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:30.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.790+0000 7f8fcf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:30.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.790+0000 7f8fcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:30.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.790+0000 7f8fcf7fe640 10 monclient(hunting): my global_id is 4371 2024-03-29T10:55:30.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.790+0000 7f8fcf7fe640 1 -- 172.21.15.112:0/1327180317 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd4004eb0 con 0x7f8fe80b2620 2024-03-29T10:55:30.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.790+0000 7f8fcf7fe640 10 client.?.objecter ms_handle_connect 0x7f8fe814fc70 2024-03-29T10:55:30.775 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.790+0000 7f8fcf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:30.775 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.791+0000 7f8fcf7fe640 1 -- 172.21.15.112:0/1327180317 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180300191 0 0) 0x7f8fe813ee30 con 0x7f8fe814fc70 2024-03-29T10:55:30.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.791+0000 7f8fcf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:30.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.791+0000 7f8fcf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:30.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.791+0000 7f8fcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:30.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.791+0000 7f8fcf7fe640 10 monclient(hunting): my global_id is 4363 2024-03-29T10:55:30.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.791+0000 7f8fcf7fe640 1 -- 172.21.15.112:0/1327180317 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd40051d0 con 0x7f8fe814fc70 2024-03-29T10:55:30.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.792+0000 7f8fcf7fe640 1 -- 172.21.15.112:0/1327180317 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2796379750 0 0) 0x7f8fd4004eb0 con 0x7f8fe80b2620 2024-03-29T10:55:30.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.792+0000 7f8fcf7fe640 1 -- 172.21.15.112:0/1327180317 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fe813ee30 con 0x7f8fe80b2620 2024-03-29T10:55:30.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.792+0000 7f8fcf7fe640 1 -- 172.21.15.112:0/1327180317 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f8fd8004040 con 0x7f8fe80b2620 2024-03-29T10:55:30.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.792+0000 7f8fcf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:30.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.792+0000 7f8fcf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:30.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.792+0000 7f8fcf7fe640 10 monclient(hunting): dump: 2024-03-29T10:55:30.776 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:30.776 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:30.777 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:30.777 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:30.777 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:30.777 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:30.777 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:30.777 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:30.777 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:30.777 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:30.777 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.792+0000 7f8fcf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:30.777 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.792+0000 7f8fcf7fe640 1 -- 172.21.15.112:0/1327180317 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2935475083 0 0) 0x7f8fd40051d0 con 0x7f8fe814fc70 2024-03-29T10:55:30.777 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.792+0000 7f8fcf7fe640 1 -- 172.21.15.112:0/1327180317 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fd4004eb0 con 0x7f8fe814fc70 2024-03-29T10:55:30.777 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.792+0000 7f8fcf7fe640 1 -- 172.21.15.112:0/1327180317 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f8fe0001a80 con 0x7f8fe814fc70 2024-03-29T10:55:30.777 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.792+0000 7f8fcf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:30.777 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.792+0000 7f8fcf7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:30.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.792+0000 7f8fcf7fe640 10 monclient(hunting): dump: 2024-03-29T10:55:30.778 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:30.778 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:30.778 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:30.778 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:30.778 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:30.778 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:30.778 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:30.778 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:30.778 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:30.778 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:30.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.793+0000 7f8fcf7fe640 10 client.?.objecter ms_handle_connect 0x7f8fe8149b50 2024-03-29T10:55:30.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.793+0000 7f8fcf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:30.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.793+0000 7f8fcf7fe640 1 -- 172.21.15.112:0/1327180317 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 562519769 0 0) 0x7f8fe813ee30 con 0x7f8fe80b2620 2024-03-29T10:55:30.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.793+0000 7f8fcf7fe640 1 -- 172.21.15.112:0/1327180317 >> v1:172.21.15.112:6790/0 conn(0x7f8fe8149b50 legacy=0x7f8fe813aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:30.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.793+0000 7f8fcf7fe640 1 -- 172.21.15.112:0/1327180317 >> v1:172.21.15.118:6789/0 conn(0x7f8fe814fc70 legacy=0x7f8fe813b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:30.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.793+0000 7f8fcf7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:55:30.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.793+0000 7f8fcf7fe640 1 monclient: found mon.a 2024-03-29T10:55:30.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.793+0000 7f8fcf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:30.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.793+0000 7f8fcf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:55:30.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.793+0000 7f8fcf7fe640 1 -- 172.21.15.112:0/1327180317 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe8144160 con 0x7f8fe80b2620 2024-03-29T10:55:30.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.793+0000 7f8fcf7fe640 10 monclient: _finish_auth 0 2024-03-29T10:55:30.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.793+0000 7f8fcf7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:55:30.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.793+0000 7f8fcf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:30.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.793+0000 7f8fed001640 5 monclient: authenticate success, global_id 4371 2024-03-29T10:55:30.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.793+0000 7f8fed001640 10 monclient: _renew_subs 2024-03-29T10:55:30.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.793+0000 7f8fed001640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:55:30.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.793+0000 7f8fed001640 1 -- 172.21.15.112:0/1327180317 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe813e940 con 0x7f8fe80b2620 2024-03-29T10:55:30.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.793+0000 7f8fed001640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:30.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.793+0000 7f8fed001640 10 monclient: _renew_subs 2024-03-29T10:55:30.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.793+0000 7f8fed001640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:55:30.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.793+0000 7f8fed001640 1 -- 172.21.15.112:0/1327180317 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe81df520 con 0x7f8fe80b2620 2024-03-29T10:55:30.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.793+0000 7f8fed001640 1 librados: init done 2024-03-29T10:55:30.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.794+0000 7f8fcf7fe640 1 -- 172.21.15.112:0/1327180317 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fd8000ed0 con 0x7f8fe80b2620 2024-03-29T10:55:30.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.794+0000 7f8fcf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:30.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.794+0000 7f8fcf7fe640 1 -- 172.21.15.112:0/1327180317 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f8fd8001260 con 0x7f8fe80b2620 2024-03-29T10:55:30.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.794+0000 7f8fcf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:30.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.794+0000 7f8fcf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:30.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.794+0000 7f8fcf7fe640 10 monclient: dump: 2024-03-29T10:55:30.780 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:30.780 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:30.780 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:30.780 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:30.781 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:30.781 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:30.781 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:30.781 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:30.781 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:30.781 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:30.781 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.794+0000 7f8fe559a640 4 set_mon_vals no callback set 2024-03-29T10:55:30.782 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.796+0000 7f8fed001640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:30.782 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.796+0000 7f8fed001640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:30.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.796+0000 7f8fed001640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:55:30.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.796+0000 7f8fed001640 1 -- 172.21.15.112:0/1327180317 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fc8005080 con 0x7f8fe80b2620 2024-03-29T10:55:30.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.799+0000 7f8fcf7fe640 1 -- 172.21.15.112:0/1327180317 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f8fd80013c0 con 0x7f8fe80b2620 2024-03-29T10:55:30.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.800+0000 7f8fcf7fe640 1 -- 172.21.15.112:0/1327180317 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f8fd80849b0 con 0x7f8fe80b2620 2024-03-29T10:55:30.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.800+0000 7f8fcf7fe640 10 client.4371.objecter ms_dispatch 0x7f8fe8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:30.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.800+0000 7f8fcf7fe640 3 client.4371.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:30.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.800+0000 7f8fcf7fe640 3 client.4371.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:30.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.800+0000 7f8fcf7fe640 20 client.4371.objecter dump_active .. 0 homeless 2024-03-29T10:55:30.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.800+0000 7f8fe659c640 10 monclient: get_auth_request con 0x7f8fd406df60 auth_method 0 2024-03-29T10:55:30.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.801+0000 7f8fcf7fe640 10 client.4371.objecter ms_handle_connect 0x7f8fd406df60 2024-03-29T10:55:30.789 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.808+0000 7f8fcf7fe640 1 -- 172.21.15.112:0/1327180317 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8fd8050290 con 0x7f8fe80b2620 2024-03-29T10:55:30.790 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.808+0000 7f8fcf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:30.790 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:30.808+0000 7f8fcf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:31.150 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:31.168+0000 7f8fed001640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:31.150 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:31.169+0000 7f8fed001640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:31.150 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:31.169+0000 7f8fed001640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:55:31.150 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:31.169+0000 7f8fed001640 1 -- 172.21.15.112:0/1327180317 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fc8005330 con 0x7f8fe80b2620 2024-03-29T10:55:31.151 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:31.170+0000 7f8fcf7fe640 1 -- 172.21.15.112:0/1327180317 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f8fd80521c0 con 0x7f8fe80b2620 2024-03-29T10:55:31.151 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:31.170+0000 7f8fcf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:31.151 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:31.170+0000 7f8fcf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:31.152 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:55:31.152 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:31.155 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:31.174+0000 7f8fed001640 10 librados: watch_flush enter 2024-03-29T10:55:31.155 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:31.174+0000 7f8fed001640 10 librados: watch_flush exit 2024-03-29T10:55:31.155 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:31.174+0000 7f8fed001640 20 client.4371.objecter shutdown clearing up homeless session... 2024-03-29T10:55:31.155 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:31.174+0000 7f8fed001640 10 client.4371.objecter successfully canceled tick 2024-03-29T10:55:31.155 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:31.174+0000 7f8fed001640 1 -- 172.21.15.112:0/1327180317 >> v1:172.21.15.118:6816/3181073408 conn(0x7f8fd406df60 legacy=0x7f8fd4070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:31.155 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:31.174+0000 7f8fed001640 10 monclient: shutdown 2024-03-29T10:55:31.155 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:31.174+0000 7f8fed001640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:31.155 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:31.174+0000 7f8fed001640 1 -- 172.21.15.112:0/1327180317 >> v1:172.21.15.112:6789/0 conn(0x7f8fe80b2620 legacy=0x7f8fe813f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:31.156 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:31.174+0000 7f8fed001640 1 -- 172.21.15.112:0/1327180317 shutdown_connections 2024-03-29T10:55:31.156 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:31.174+0000 7f8fed001640 1 -- 172.21.15.112:0/1327180317 >> 172.21.15.112:0/1327180317 conn(0x7f8fe806d180 msgr2=0x7f8fe80b2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:31.156 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:31.174+0000 7f8fed001640 1 -- 172.21.15.112:0/1327180317 shutdown_connections 2024-03-29T10:55:31.156 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:31.174+0000 7f8fed001640 1 -- 172.21.15.112:0/1327180317 wait complete. 2024-03-29T10:55:31.156 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:31.175+0000 7f8fed001640 1 librados: shutdown 2024-03-29T10:55:31.173 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:37.980 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:38.131 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.150+0000 7f2e7007b640 10 monclient: get_monmap_and_config 2024-03-29T10:55:38.131 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.150+0000 7f2e7007b640 10 monclient: build_initial_monmap 2024-03-29T10:55:38.131 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.150+0000 7f2e7007b640 10 monclient: monmap: 2024-03-29T10:55:38.131 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:55:38.131 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:38.131 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:55:38.150991+0000 2024-03-29T10:55:38.131 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:55:38.150991+0000 2024-03-29T10:55:38.131 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:38.131 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:38.131 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:55:38.132 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:55:38.132 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:55:38.132 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:38.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.150+0000 7f2e7007b640 1 Processor -- start 2024-03-29T10:55:38.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.151+0000 7f2e7007b640 1 -- start start 2024-03-29T10:55:38.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.151+0000 7f2e7007b640 10 monclient: init 2024-03-29T10:55:38.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.152+0000 7f2e7007b640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:38.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.152+0000 7f2e7007b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:55:38.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.152+0000 7f2e7007b640 10 monclient(hunting): picked mon.noname-c con 0x7f2e681457f0 addr v1:172.21.15.118:6789/0 2024-03-29T10:55:38.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.152+0000 7f2e7007b640 10 monclient(hunting): picked mon.noname-a con 0x7f2e6813f650 addr v1:172.21.15.112:6789/0 2024-03-29T10:55:38.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.152+0000 7f2e7007b640 10 monclient(hunting): picked mon.noname-b con 0x7f2e680a9d70 addr v1:172.21.15.112:6790/0 2024-03-29T10:55:38.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.152+0000 7f2e7007b640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e680aaef0 con 0x7f2e6813f650 2024-03-29T10:55:38.136 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.152+0000 7f2e7007b640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e68145bd0 con 0x7f2e681457f0 2024-03-29T10:55:38.136 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.152+0000 7f2e7007b640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e68145eb0 con 0x7f2e680a9d70 2024-03-29T10:55:38.136 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.152+0000 7f2e7007b640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:38.136 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.152+0000 7f2e7007b640 10 monclient(hunting): authenticate will time out at 1093.456495s 2024-03-29T10:55:38.136 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.152+0000 7f2e6e618640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f2e680a9d70 0x7f2e6814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37522/0 (socket says 172.21.15.112:37522) 2024-03-29T10:55:38.136 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.152+0000 7f2e6e618640 1 -- 172.21.15.112:0/1887152139 learned_addr learned my addr 172.21.15.112:0/1887152139 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:55:38.136 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.152+0000 7f2e6ce15640 1 -- 172.21.15.112:0/1887152139 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745419005 0 0) 0x7f2e68145eb0 con 0x7f2e680a9d70 2024-03-29T10:55:38.136 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.152+0000 7f2e6ce15640 10 monclient(hunting): _negotiate 2024-03-29T10:55:38.136 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.152+0000 7f2e6ce15640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:38.136 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.152+0000 7f2e6ce15640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:38.136 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.152+0000 7f2e6ce15640 10 monclient(hunting): my global_id is 4388 2024-03-29T10:55:38.136 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.152+0000 7f2e6ce15640 1 -- 172.21.15.112:0/1887152139 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e50003520 con 0x7f2e680a9d70 2024-03-29T10:55:38.136 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 1 -- 172.21.15.112:0/1887152139 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2892417560 0 0) 0x7f2e68145bd0 con 0x7f2e681457f0 2024-03-29T10:55:38.137 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 10 monclient(hunting): _negotiate 2024-03-29T10:55:38.137 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:38.137 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:38.137 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 10 monclient(hunting): my global_id is 4366 2024-03-29T10:55:38.137 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 1 -- 172.21.15.112:0/1887152139 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e68145eb0 con 0x7f2e681457f0 2024-03-29T10:55:38.137 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 1 -- 172.21.15.112:0/1887152139 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2188622678 0 0) 0x7f2e680aaef0 con 0x7f2e6813f650 2024-03-29T10:55:38.137 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 10 monclient(hunting): _negotiate 2024-03-29T10:55:38.137 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:38.137 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:38.137 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 10 monclient(hunting): my global_id is 4374 2024-03-29T10:55:38.137 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 1 -- 172.21.15.112:0/1887152139 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e68145bd0 con 0x7f2e6813f650 2024-03-29T10:55:38.137 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 1 -- 172.21.15.112:0/1887152139 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1856366252 0 0) 0x7f2e50003520 con 0x7f2e680a9d70 2024-03-29T10:55:38.137 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 1 -- 172.21.15.112:0/1887152139 >> v1:172.21.15.118:6789/0 conn(0x7f2e681457f0 legacy=0x7f2e6813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:38.137 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 1 -- 172.21.15.112:0/1887152139 >> v1:172.21.15.112:6789/0 conn(0x7f2e6813f650 legacy=0x7f2e680b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:38.138 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 10 monclient: _finish_hunting 0 2024-03-29T10:55:38.138 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 1 monclient: found mon.noname-b 2024-03-29T10:55:38.138 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:38.138 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:55:38.138 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 1 -- 172.21.15.112:0/1887152139 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e68146340 con 0x7f2e680a9d70 2024-03-29T10:55:38.138 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 10 monclient: _finish_auth 0 2024-03-29T10:55:38.138 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 10 monclient: _check_auth_tickets 2024-03-29T10:55:38.138 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:38.138 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 1 -- 172.21.15.112:0/1887152139 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f2e60001990 con 0x7f2e680a9d70 2024-03-29T10:55:38.138 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:38.138 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:38.138 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e6ce15640 10 monclient: dump: 2024-03-29T10:55:38.138 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:38.139 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:38.139 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:38.139 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:38.139 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:38.139 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:38.139 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:38.139 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:38.139 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:38.139 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:38.139 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e7007b640 5 monclient: authenticate success, global_id 4388 2024-03-29T10:55:38.139 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.153+0000 7f2e7007b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:38.139 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.154+0000 7f2e6ce15640 1 -- 172.21.15.112:0/1887152139 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e60001af0 con 0x7f2e680a9d70 2024-03-29T10:55:38.139 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.154+0000 7f2e6ce15640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:38.139 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.154+0000 7f2e6ce15640 1 -- 172.21.15.112:0/1887152139 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f2e60001f40 con 0x7f2e680a9d70 2024-03-29T10:55:38.140 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.154+0000 7f2e6ce15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:38.140 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.154+0000 7f2e6ce15640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:38.140 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.154+0000 7f2e6ce15640 10 monclient: dump: 2024-03-29T10:55:38.140 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:38.140 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:38.140 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:38.140 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:38.140 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:38.140 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:38.140 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:38.140 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:38.140 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:38.140 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:38.141 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.154+0000 7f2e7007b640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:38.141 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.154+0000 7f2e7007b640 4 set_mon_vals no callback set 2024-03-29T10:55:38.145 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.155+0000 7f2e7007b640 10 monclient: shutdown 2024-03-29T10:55:38.145 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.155+0000 7f2e7007b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:38.145 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.155+0000 7f2e7007b640 1 -- 172.21.15.112:0/1887152139 >> v1:172.21.15.112:6790/0 conn(0x7f2e680a9d70 legacy=0x7f2e6814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:38.145 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.155+0000 7f2e7007b640 1 -- 172.21.15.112:0/1887152139 shutdown_connections 2024-03-29T10:55:38.145 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.155+0000 7f2e7007b640 1 -- 172.21.15.112:0/1887152139 >> 172.21.15.112:0/1887152139 conn(0x7f2e68066a00 msgr2=0x7f2e68066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:38.145 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.155+0000 7f2e7007b640 1 -- 172.21.15.112:0/1887152139 shutdown_connections 2024-03-29T10:55:38.146 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.155+0000 7f2e7007b640 1 -- 172.21.15.112:0/1887152139 wait complete. 2024-03-29T10:55:38.146 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.156+0000 7f2e7007b640 10 monclient: build_initial_monmap 2024-03-29T10:55:38.146 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.156+0000 7f2e7007b640 10 monclient: monmap: 2024-03-29T10:55:38.146 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:55:38.146 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:38.146 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:55:38.146 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:55:38.146 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:38.146 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:38.146 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:55:38.147 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:55:38.147 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:55:38.147 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:38.147 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.156+0000 7f2e7007b640 1 librados: starting msgr at 2024-03-29T10:55:38.147 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.156+0000 7f2e7007b640 1 librados: starting objecter 2024-03-29T10:55:38.147 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.157+0000 7f2e7007b640 1 Processor -- start 2024-03-29T10:55:38.147 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.157+0000 7f2e7007b640 1 -- start start 2024-03-29T10:55:38.147 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.157+0000 7f2e7007b640 1 librados: setting wanted keys 2024-03-29T10:55:38.147 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.157+0000 7f2e7007b640 1 librados: calling monclient init 2024-03-29T10:55:38.147 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.157+0000 7f2e7007b640 10 monclient: init 2024-03-29T10:55:38.147 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.158+0000 7f2e7007b640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:38.147 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.158+0000 7f2e7007b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:55:38.147 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.158+0000 7f2e7007b640 10 monclient(hunting): picked mon.noname-b con 0x7f2e680a9d70 addr v1:172.21.15.118:6789/0 2024-03-29T10:55:38.148 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.158+0000 7f2e7007b640 10 monclient(hunting): picked mon.noname-a con 0x7f2e6813f650 addr v1:172.21.15.112:6789/0 2024-03-29T10:55:38.148 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.158+0000 7f2e7007b640 10 monclient(hunting): picked mon.noname-c con 0x7f2e681457f0 addr v1:172.21.15.112:6790/0 2024-03-29T10:55:38.148 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.158+0000 7f2e7007b640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e681cca90 con 0x7f2e6813f650 2024-03-29T10:55:38.148 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.158+0000 7f2e7007b640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e681cbe80 con 0x7f2e680a9d70 2024-03-29T10:55:38.148 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.158+0000 7f2e7007b640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e681ccd00 con 0x7f2e681457f0 2024-03-29T10:55:38.148 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.158+0000 7f2e7007b640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:38.148 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.158+0000 7f2e7007b640 10 monclient(hunting): authenticate will time out at 1093.462574s 2024-03-29T10:55:38.148 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.158+0000 7f2e6d616640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f2e6813f650 0x7f2e681cd400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:51370/0 (socket says 172.21.15.112:51370) 2024-03-29T10:55:38.148 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.158+0000 7f2e6d616640 1 -- 172.21.15.112:0/3620789078 learned_addr learned my addr 172.21.15.112:0/3620789078 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:55:38.148 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.158+0000 7f2e667fc640 10 client.?.objecter ms_handle_connect 0x7f2e6813f650 2024-03-29T10:55:38.148 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.158+0000 7f2e667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:38.148 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.158+0000 7f2e667fc640 1 -- 172.21.15.112:0/3620789078 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3593050209 0 0) 0x7f2e681cca90 con 0x7f2e6813f650 2024-03-29T10:55:38.148 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.158+0000 7f2e667fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:38.148 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.158+0000 7f2e667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:38.149 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.158+0000 7f2e667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:38.149 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.158+0000 7f2e667fc640 10 monclient(hunting): my global_id is 4377 2024-03-29T10:55:38.149 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.158+0000 7f2e667fc640 1 -- 172.21.15.112:0/3620789078 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e54004eb0 con 0x7f2e6813f650 2024-03-29T10:55:38.149 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.159+0000 7f2e667fc640 10 client.?.objecter ms_handle_connect 0x7f2e681457f0 2024-03-29T10:55:38.149 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.159+0000 7f2e667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:38.149 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.159+0000 7f2e667fc640 10 client.?.objecter ms_handle_connect 0x7f2e680a9d70 2024-03-29T10:55:38.149 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.159+0000 7f2e667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:38.149 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.159+0000 7f2e667fc640 1 -- 172.21.15.112:0/3620789078 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543614748 0 0) 0x7f2e681ccd00 con 0x7f2e681457f0 2024-03-29T10:55:38.149 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.159+0000 7f2e667fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:38.149 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.159+0000 7f2e667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:38.150 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.159+0000 7f2e667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:38.150 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.159+0000 7f2e667fc640 10 monclient(hunting): my global_id is 4391 2024-03-29T10:55:38.150 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.159+0000 7f2e667fc640 1 -- 172.21.15.112:0/3620789078 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e681cca90 con 0x7f2e681457f0 2024-03-29T10:55:38.150 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.159+0000 7f2e667fc640 1 -- 172.21.15.112:0/3620789078 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1509262753 0 0) 0x7f2e681cbe80 con 0x7f2e680a9d70 2024-03-29T10:55:38.150 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.159+0000 7f2e667fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:38.150 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.159+0000 7f2e667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:38.150 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.159+0000 7f2e667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:38.150 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.159+0000 7f2e667fc640 10 monclient(hunting): my global_id is 4369 2024-03-29T10:55:38.150 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.159+0000 7f2e667fc640 1 -- 172.21.15.112:0/3620789078 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e54005360 con 0x7f2e680a9d70 2024-03-29T10:55:38.150 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 1 -- 172.21.15.112:0/3620789078 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1393479780 0 0) 0x7f2e54004eb0 con 0x7f2e6813f650 2024-03-29T10:55:38.150 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 1 -- 172.21.15.112:0/3620789078 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e540057b0 con 0x7f2e6813f650 2024-03-29T10:55:38.150 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 1 -- 172.21.15.112:0/3620789078 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f2e5c001a10 con 0x7f2e6813f650 2024-03-29T10:55:38.150 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:38.151 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:38.151 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 10 monclient(hunting): dump: 2024-03-29T10:55:38.151 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:38.151 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:38.151 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:38.151 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:38.151 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:38.151 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:38.151 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:38.151 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:38.151 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:38.151 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:38.151 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:38.152 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 1 -- 172.21.15.112:0/3620789078 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1574958776 0 0) 0x7f2e681cca90 con 0x7f2e681457f0 2024-03-29T10:55:38.152 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 1 -- 172.21.15.112:0/3620789078 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e54004eb0 con 0x7f2e681457f0 2024-03-29T10:55:38.152 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 1 -- 172.21.15.112:0/3620789078 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f2e60001c60 con 0x7f2e681457f0 2024-03-29T10:55:38.152 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:38.152 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:38.152 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 10 monclient(hunting): dump: 2024-03-29T10:55:38.152 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:38.152 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:38.152 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:38.152 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:38.152 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:38.152 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:38.152 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:38.153 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:38.153 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:38.153 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:38.153 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 1 -- 172.21.15.112:0/3620789078 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3536561853 0 0) 0x7f2e540057b0 con 0x7f2e6813f650 2024-03-29T10:55:38.153 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 1 -- 172.21.15.112:0/3620789078 >> v1:172.21.15.112:6790/0 conn(0x7f2e681457f0 legacy=0x7f2e681e2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:38.153 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 1 -- 172.21.15.112:0/3620789078 >> v1:172.21.15.118:6789/0 conn(0x7f2e680a9d70 legacy=0x7f2e68137d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:38.153 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 10 monclient: _finish_hunting 0 2024-03-29T10:55:38.153 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 1 monclient: found mon.a 2024-03-29T10:55:38.153 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:38.153 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:55:38.153 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 1 -- 172.21.15.112:0/3620789078 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e681ccfe0 con 0x7f2e6813f650 2024-03-29T10:55:38.153 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 10 monclient: _finish_auth 0 2024-03-29T10:55:38.153 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 10 monclient: _check_auth_tickets 2024-03-29T10:55:38.154 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:38.154 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e7007b640 5 monclient: authenticate success, global_id 4377 2024-03-29T10:55:38.154 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e7007b640 10 monclient: _renew_subs 2024-03-29T10:55:38.154 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e7007b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:55:38.154 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e7007b640 1 -- 172.21.15.112:0/3620789078 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e681cb960 con 0x7f2e6813f650 2024-03-29T10:55:38.154 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 1 -- 172.21.15.112:0/3620789078 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e5c001c50 con 0x7f2e6813f650 2024-03-29T10:55:38.154 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:38.154 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 1 -- 172.21.15.112:0/3620789078 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f2e5c003a60 con 0x7f2e6813f650 2024-03-29T10:55:38.154 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:38.154 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:38.154 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e667fc640 10 monclient: dump: 2024-03-29T10:55:38.154 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:38.154 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:38.155 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:38.155 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:38.155 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:38.155 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:38.155 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:38.155 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:38.155 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:38.155 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:38.155 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e67fff640 4 set_mon_vals no callback set 2024-03-29T10:55:38.155 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e7007b640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:38.155 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e7007b640 10 monclient: _renew_subs 2024-03-29T10:55:38.155 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e7007b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:55:38.155 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e7007b640 1 -- 172.21.15.112:0/3620789078 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e68142b30 con 0x7f2e6813f650 2024-03-29T10:55:38.156 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.160+0000 7f2e7007b640 1 librados: init done 2024-03-29T10:55:38.156 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.162+0000 7f2e7007b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:38.156 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.162+0000 7f2e7007b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:38.156 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.162+0000 7f2e7007b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:55:38.156 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.162+0000 7f2e7007b640 1 -- 172.21.15.112:0/3620789078 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e68075f10 con 0x7f2e6813f650 2024-03-29T10:55:38.156 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.164+0000 7f2e667fc640 1 -- 172.21.15.112:0/3620789078 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f2e5c003bc0 con 0x7f2e6813f650 2024-03-29T10:55:38.156 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.164+0000 7f2e667fc640 1 -- 172.21.15.112:0/3620789078 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f2e5c084b10 con 0x7f2e6813f650 2024-03-29T10:55:38.156 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.164+0000 7f2e667fc640 10 client.4377.objecter ms_dispatch 0x7f2e68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:38.156 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.164+0000 7f2e667fc640 3 client.4377.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:38.156 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.164+0000 7f2e667fc640 3 client.4377.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:38.156 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.164+0000 7f2e6de17640 10 monclient: get_auth_request con 0x7f2e5406e390 auth_method 0 2024-03-29T10:55:38.156 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.164+0000 7f2e667fc640 20 client.4377.objecter dump_active .. 0 homeless 2024-03-29T10:55:38.157 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.165+0000 7f2e667fc640 10 client.4377.objecter ms_handle_connect 0x7f2e5406e390 2024-03-29T10:55:38.157 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.173+0000 7f2e667fc640 1 -- 172.21.15.112:0/3620789078 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e5c0503f0 con 0x7f2e6813f650 2024-03-29T10:55:38.157 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.173+0000 7f2e667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:38.157 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.173+0000 7f2e667fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:38.530 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.549+0000 7f2e7007b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:38.530 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.549+0000 7f2e7007b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:38.530 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.549+0000 7f2e7007b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:55:38.530 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.549+0000 7f2e7007b640 1 -- 172.21.15.112:0/3620789078 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e680b3680 con 0x7f2e6813f650 2024-03-29T10:55:38.531 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.550+0000 7f2e667fc640 1 -- 172.21.15.112:0/3620789078 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f2e5c052320 con 0x7f2e6813f650 2024-03-29T10:55:38.531 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.550+0000 7f2e667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:38.531 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.550+0000 7f2e667fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:38.531 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:55:38.531 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:38.535 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.553+0000 7f2e7007b640 10 librados: watch_flush enter 2024-03-29T10:55:38.535 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.554+0000 7f2e7007b640 10 librados: watch_flush exit 2024-03-29T10:55:38.535 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.554+0000 7f2e7007b640 20 client.4377.objecter shutdown clearing up homeless session... 2024-03-29T10:55:38.535 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.554+0000 7f2e7007b640 10 client.4377.objecter successfully canceled tick 2024-03-29T10:55:38.535 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.554+0000 7f2e7007b640 1 -- 172.21.15.112:0/3620789078 >> v1:172.21.15.118:6816/3181073408 conn(0x7f2e5406e390 legacy=0x7f2e54070850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:38.535 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.554+0000 7f2e7007b640 10 monclient: shutdown 2024-03-29T10:55:38.535 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.554+0000 7f2e7007b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:38.535 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.554+0000 7f2e7007b640 1 -- 172.21.15.112:0/3620789078 >> v1:172.21.15.112:6789/0 conn(0x7f2e6813f650 legacy=0x7f2e681cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:38.535 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.554+0000 7f2e7007b640 1 -- 172.21.15.112:0/3620789078 shutdown_connections 2024-03-29T10:55:38.536 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.554+0000 7f2e7007b640 1 -- 172.21.15.112:0/3620789078 >> 172.21.15.112:0/3620789078 conn(0x7f2e68066a00 msgr2=0x7f2e680b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:38.536 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.554+0000 7f2e7007b640 1 -- 172.21.15.112:0/3620789078 shutdown_connections 2024-03-29T10:55:38.536 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.554+0000 7f2e7007b640 1 -- 172.21.15.112:0/3620789078 wait complete. 2024-03-29T10:55:38.536 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:38.554+0000 7f2e7007b640 1 librados: shutdown 2024-03-29T10:55:38.549 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:45.356 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:45.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.502+0000 7f9a107f6640 10 monclient: get_monmap_and_config 2024-03-29T10:55:45.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.502+0000 7f9a107f6640 10 monclient: build_initial_monmap 2024-03-29T10:55:45.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.502+0000 7f9a107f6640 10 monclient: monmap: 2024-03-29T10:55:45.484 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:55:45.484 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:45.484 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:55:45.503815+0000 2024-03-29T10:55:45.484 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:55:45.503815+0000 2024-03-29T10:55:45.484 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:45.484 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:45.484 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:55:45.484 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:55:45.484 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:55:45.484 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:45.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.503+0000 7f9a107f6640 1 Processor -- start 2024-03-29T10:55:45.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.503+0000 7f9a107f6640 1 -- start start 2024-03-29T10:55:45.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.503+0000 7f9a107f6640 10 monclient: init 2024-03-29T10:55:45.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.504+0000 7f9a107f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:45.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.504+0000 7f9a107f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:55:45.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.504+0000 7f9a107f6640 10 monclient(hunting): picked mon.noname-c con 0x7f9a081457f0 addr v1:172.21.15.118:6789/0 2024-03-29T10:55:45.486 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.505+0000 7f9a107f6640 10 monclient(hunting): picked mon.noname-b con 0x7f9a0813f650 addr v1:172.21.15.112:6790/0 2024-03-29T10:55:45.486 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.505+0000 7f9a107f6640 10 monclient(hunting): picked mon.noname-a con 0x7f9a080a9d70 addr v1:172.21.15.112:6789/0 2024-03-29T10:55:45.486 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.505+0000 7f9a107f6640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a080aaef0 con 0x7f9a080a9d70 2024-03-29T10:55:45.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.505+0000 7f9a107f6640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a08145bd0 con 0x7f9a081457f0 2024-03-29T10:55:45.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.505+0000 7f9a107f6640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a08145eb0 con 0x7f9a0813f650 2024-03-29T10:55:45.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.505+0000 7f9a107f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:45.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.505+0000 7f9a107f6640 10 monclient(hunting): authenticate will time out at 1100.809432s 2024-03-29T10:55:45.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.505+0000 7f9a0ed93640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f9a080a9d70 0x7f9a0814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:41084/0 (socket says 172.21.15.112:41084) 2024-03-29T10:55:45.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.505+0000 7f9a0ed93640 1 -- 172.21.15.112:0/2069185304 learned_addr learned my addr 172.21.15.112:0/2069185304 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:55:45.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.505+0000 7f9a0d590640 1 -- 172.21.15.112:0/2069185304 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 554093623 0 0) 0x7f9a080aaef0 con 0x7f9a080a9d70 2024-03-29T10:55:45.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.505+0000 7f9a0d590640 10 monclient(hunting): _negotiate 2024-03-29T10:55:45.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.505+0000 7f9a0d590640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:45.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.505+0000 7f9a0d590640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:45.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.505+0000 7f9a0d590640 10 monclient(hunting): my global_id is 4380 2024-03-29T10:55:45.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.505+0000 7f9a0d590640 1 -- 172.21.15.112:0/2069185304 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f4003520 con 0x7f9a080a9d70 2024-03-29T10:55:45.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.506+0000 7f9a0d590640 1 -- 172.21.15.112:0/2069185304 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2077821480 0 0) 0x7f99f4003520 con 0x7f9a080a9d70 2024-03-29T10:55:45.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.506+0000 7f9a0d590640 1 -- 172.21.15.112:0/2069185304 >> v1:172.21.15.112:6790/0 conn(0x7f9a0813f650 legacy=0x7f9a080b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:45.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.507+0000 7f9a0d590640 1 -- 172.21.15.112:0/2069185304 >> v1:172.21.15.118:6789/0 conn(0x7f9a081457f0 legacy=0x7f9a0813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:45.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.507+0000 7f9a0d590640 10 monclient: _finish_hunting 0 2024-03-29T10:55:45.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.507+0000 7f9a0d590640 1 monclient: found mon.noname-a 2024-03-29T10:55:45.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.507+0000 7f9a0d590640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:45.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.507+0000 7f9a0d590640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T10:55:45.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.507+0000 7f9a0d590640 1 -- 172.21.15.112:0/2069185304 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a08146340 con 0x7f9a080a9d70 2024-03-29T10:55:45.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.507+0000 7f9a0d590640 10 monclient: _finish_auth 0 2024-03-29T10:55:45.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.507+0000 7f9a0d590640 10 monclient: _check_auth_tickets 2024-03-29T10:55:45.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.507+0000 7f9a0d590640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:45.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.507+0000 7f9a0d590640 1 -- 172.21.15.112:0/2069185304 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9a00001a10 con 0x7f9a080a9d70 2024-03-29T10:55:45.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.507+0000 7f9a0d590640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:45.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.507+0000 7f9a0d590640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:45.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.507+0000 7f9a0d590640 10 monclient: dump: 2024-03-29T10:55:45.489 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:45.489 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:45.489 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:45.489 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:45.490 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:45.490 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:45.490 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:45.490 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:45.490 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:45.490 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:45.490 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.507+0000 7f9a107f6640 5 monclient: authenticate success, global_id 4380 2024-03-29T10:55:45.490 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.507+0000 7f9a107f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:45.491 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.508+0000 7f9a0d590640 1 -- 172.21.15.112:0/2069185304 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a00001be0 con 0x7f9a080a9d70 2024-03-29T10:55:45.491 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.508+0000 7f9a0d590640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:45.491 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.508+0000 7f9a0d590640 1 -- 172.21.15.112:0/2069185304 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9a00002030 con 0x7f9a080a9d70 2024-03-29T10:55:45.491 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.508+0000 7f9a107f6640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:45.491 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.508+0000 7f9a107f6640 4 set_mon_vals no callback set 2024-03-29T10:55:45.492 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.508+0000 7f9a0d590640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:45.492 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.508+0000 7f9a0d590640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:45.492 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.508+0000 7f9a0d590640 10 monclient: dump: 2024-03-29T10:55:45.492 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:45.492 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:45.492 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:45.492 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:45.492 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:45.492 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:45.492 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:45.492 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:45.492 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:45.492 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:45.492 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.508+0000 7f9a107f6640 10 monclient: shutdown 2024-03-29T10:55:45.493 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.509+0000 7f9a107f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:45.493 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.509+0000 7f9a107f6640 1 -- 172.21.15.112:0/2069185304 >> v1:172.21.15.112:6789/0 conn(0x7f9a080a9d70 legacy=0x7f9a0814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:45.493 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.509+0000 7f9a107f6640 1 -- 172.21.15.112:0/2069185304 shutdown_connections 2024-03-29T10:55:45.493 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.509+0000 7f9a107f6640 1 -- 172.21.15.112:0/2069185304 >> 172.21.15.112:0/2069185304 conn(0x7f9a08066a00 msgr2=0x7f9a08066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:45.493 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.509+0000 7f9a107f6640 1 -- 172.21.15.112:0/2069185304 shutdown_connections 2024-03-29T10:55:45.493 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.510+0000 7f9a107f6640 1 -- 172.21.15.112:0/2069185304 wait complete. 2024-03-29T10:55:45.493 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.511+0000 7f9a107f6640 10 monclient: build_initial_monmap 2024-03-29T10:55:45.493 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.511+0000 7f9a107f6640 10 monclient: monmap: 2024-03-29T10:55:45.493 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:55:45.493 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:45.493 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:55:45.493 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:55:45.493 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:45.493 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:45.493 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:55:45.494 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:55:45.494 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:55:45.494 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:45.500 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.511+0000 7f9a107f6640 1 librados: starting msgr at 2024-03-29T10:55:45.500 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.511+0000 7f9a107f6640 1 librados: starting objecter 2024-03-29T10:55:45.500 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.511+0000 7f9a107f6640 1 Processor -- start 2024-03-29T10:55:45.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.512+0000 7f9a107f6640 1 -- start start 2024-03-29T10:55:45.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.512+0000 7f9a107f6640 1 librados: setting wanted keys 2024-03-29T10:55:45.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.512+0000 7f9a107f6640 1 librados: calling monclient init 2024-03-29T10:55:45.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.512+0000 7f9a107f6640 10 monclient: init 2024-03-29T10:55:45.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.512+0000 7f9a107f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:45.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.512+0000 7f9a107f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:55:45.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.512+0000 7f9a107f6640 10 monclient(hunting): picked mon.noname-a con 0x7f9a080a9d70 addr v1:172.21.15.112:6789/0 2024-03-29T10:55:45.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.513+0000 7f9a0e592640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f9a080a9d70 0x7f9a081394d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:43306/0 (socket says 172.21.15.112:43306) 2024-03-29T10:55:45.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.513+0000 7f9a0e592640 1 -- 172.21.15.112:0/4243573595 learned_addr learned my addr 172.21.15.112:0/4243573595 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:55:45.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.513+0000 7f9a107f6640 10 monclient(hunting): picked mon.noname-c con 0x7f9a0813f650 addr v1:172.21.15.112:6790/0 2024-03-29T10:55:45.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.513+0000 7f9a107f6640 10 monclient(hunting): picked mon.noname-b con 0x7f9a081457f0 addr v1:172.21.15.118:6789/0 2024-03-29T10:55:45.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.513+0000 7f9a107f6640 1 -- 172.21.15.112:0/4243573595 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a0813ae00 con 0x7f9a080a9d70 2024-03-29T10:55:45.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.513+0000 7f9a107f6640 1 -- 172.21.15.112:0/4243573595 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a08138e70 con 0x7f9a081457f0 2024-03-29T10:55:45.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.513+0000 7f9a107f6640 1 -- 172.21.15.112:0/4243573595 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a081d8390 con 0x7f9a0813f650 2024-03-29T10:55:45.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.513+0000 7f9a107f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:45.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.513+0000 7f9a107f6640 10 monclient(hunting): authenticate will time out at 1100.817950s 2024-03-29T10:55:45.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.513+0000 7f99feffd640 10 client.?.objecter ms_handle_connect 0x7f9a080a9d70 2024-03-29T10:55:45.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.513+0000 7f99feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:45.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.513+0000 7f99feffd640 1 -- 172.21.15.112:0/4243573595 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699974759 0 0) 0x7f9a0813ae00 con 0x7f9a080a9d70 2024-03-29T10:55:45.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.513+0000 7f99feffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:45.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.513+0000 7f99feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:45.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.513+0000 7f99feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:45.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.513+0000 7f99feffd640 10 monclient(hunting): my global_id is 4383 2024-03-29T10:55:45.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.513+0000 7f99feffd640 1 -- 172.21.15.112:0/4243573595 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99ec004eb0 con 0x7f9a080a9d70 2024-03-29T10:55:45.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.513+0000 7f99feffd640 10 client.?.objecter ms_handle_connect 0x7f9a081457f0 2024-03-29T10:55:45.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.513+0000 7f99feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:45.503 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.514+0000 7f99feffd640 1 -- 172.21.15.112:0/4243573595 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 533383003 0 0) 0x7f99ec004eb0 con 0x7f9a080a9d70 2024-03-29T10:55:45.503 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.514+0000 7f99feffd640 1 -- 172.21.15.112:0/4243573595 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a0813ae00 con 0x7f9a080a9d70 2024-03-29T10:55:45.503 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.514+0000 7f99feffd640 1 -- 172.21.15.112:0/4243573595 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9a04001970 con 0x7f9a080a9d70 2024-03-29T10:55:45.503 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.514+0000 7f99feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:45.503 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.514+0000 7f99feffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:45.503 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.514+0000 7f99feffd640 10 monclient(hunting): dump: 2024-03-29T10:55:45.503 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:45.503 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:45.503 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:45.503 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:45.503 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:45.503 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:45.503 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:45.503 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:45.504 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:45.504 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:45.504 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.514+0000 7f99feffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:45.504 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.514+0000 7f99feffd640 1 -- 172.21.15.112:0/4243573595 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1698026689 0 0) 0x7f9a0813ae00 con 0x7f9a080a9d70 2024-03-29T10:55:45.504 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.514+0000 7f99feffd640 1 -- 172.21.15.112:0/4243573595 >> v1:172.21.15.112:6790/0 conn(0x7f9a0813f650 legacy=0x7f9a081d7570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:45.504 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.515+0000 7f99feffd640 1 -- 172.21.15.112:0/4243573595 >> v1:172.21.15.118:6789/0 conn(0x7f9a081457f0 legacy=0x7f9a081d7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:45.504 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.515+0000 7f99feffd640 10 monclient: _finish_hunting 0 2024-03-29T10:55:45.504 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.515+0000 7f99feffd640 1 monclient: found mon.a 2024-03-29T10:55:45.504 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.515+0000 7f99feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:45.504 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.515+0000 7f99feffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:55:45.504 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.515+0000 7f99feffd640 1 -- 172.21.15.112:0/4243573595 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a080b1670 con 0x7f9a080a9d70 2024-03-29T10:55:45.504 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.515+0000 7f99feffd640 10 monclient: _finish_auth 0 2024-03-29T10:55:45.504 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.515+0000 7f99feffd640 10 monclient: _check_auth_tickets 2024-03-29T10:55:45.504 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.515+0000 7f99feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:45.505 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.515+0000 7f9a107f6640 5 monclient: authenticate success, global_id 4383 2024-03-29T10:55:45.505 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.515+0000 7f9a107f6640 10 monclient: _renew_subs 2024-03-29T10:55:45.505 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.515+0000 7f9a107f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:55:45.505 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.515+0000 7f9a107f6640 1 -- 172.21.15.112:0/4243573595 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a08138a90 con 0x7f9a080a9d70 2024-03-29T10:55:45.505 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.515+0000 7f9a107f6640 10 client.4383.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:45.505 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.515+0000 7f9a107f6640 10 monclient: _renew_subs 2024-03-29T10:55:45.505 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.515+0000 7f9a107f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:55:45.505 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.515+0000 7f9a107f6640 1 -- 172.21.15.112:0/4243573595 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a08142d80 con 0x7f9a080a9d70 2024-03-29T10:55:45.505 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.515+0000 7f9a107f6640 1 librados: init done 2024-03-29T10:55:45.505 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.515+0000 7f99feffd640 1 -- 172.21.15.112:0/4243573595 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a04001b90 con 0x7f9a080a9d70 2024-03-29T10:55:45.505 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.516+0000 7f99feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:45.505 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.516+0000 7f99feffd640 1 -- 172.21.15.112:0/4243573595 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9a04003a60 con 0x7f9a080a9d70 2024-03-29T10:55:45.505 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.516+0000 7f99feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:45.505 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.516+0000 7f99feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:45.506 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.516+0000 7f99feffd640 10 monclient: dump: 2024-03-29T10:55:45.506 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:45.506 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:45.506 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:45.506 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:45.506 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:45.506 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:45.506 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:45.506 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:45.506 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:45.506 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:45.506 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.516+0000 7f9a0cd8f640 4 set_mon_vals no callback set 2024-03-29T10:55:45.506 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.517+0000 7f99fcff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:45.506 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.517+0000 7f99fcff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:45.507 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.517+0000 7f99fcff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:55:45.507 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.517+0000 7f99fcff9640 1 -- 172.21.15.112:0/4243573595 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a08139de0 con 0x7f9a080a9d70 2024-03-29T10:55:45.507 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.518+0000 7f99feffd640 1 -- 172.21.15.112:0/4243573595 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f9a04003bc0 con 0x7f9a080a9d70 2024-03-29T10:55:45.507 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.519+0000 7f9a0dd91640 10 monclient: get_auth_request con 0x7f99ec06dbd0 auth_method 0 2024-03-29T10:55:45.507 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.519+0000 7f99feffd640 1 -- 172.21.15.112:0/4243573595 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f9a04084b20 con 0x7f9a080a9d70 2024-03-29T10:55:45.507 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.519+0000 7f99feffd640 10 client.4383.objecter ms_dispatch 0x7f9a08000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:45.507 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.519+0000 7f99feffd640 3 client.4383.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:45.507 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.519+0000 7f99feffd640 3 client.4383.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:45.507 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.519+0000 7f99feffd640 20 client.4383.objecter dump_active .. 0 homeless 2024-03-29T10:55:45.507 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.520+0000 7f99feffd640 10 client.4383.objecter ms_handle_connect 0x7f99ec06dbd0 2024-03-29T10:55:45.508 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.525+0000 7f99feffd640 1 -- 172.21.15.112:0/4243573595 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a04050230 con 0x7f9a080a9d70 2024-03-29T10:55:45.508 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.525+0000 7f99feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:45.508 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.525+0000 7f99feffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:45.815 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.834+0000 7f99fcff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:45.815 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.834+0000 7f99fcff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:45.815 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.834+0000 7f99fcff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:55:45.815 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.834+0000 7f99fcff9640 1 -- 172.21.15.112:0/4243573595 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a08075f10 con 0x7f9a080a9d70 2024-03-29T10:55:45.815 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.834+0000 7f99feffd640 1 -- 172.21.15.112:0/4243573595 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f9a04052330 con 0x7f9a080a9d70 2024-03-29T10:55:45.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.834+0000 7f99feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:45.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.834+0000 7f99feffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:45.816 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:55:45.816 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:45.819 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.837+0000 7f99fcff9640 10 librados: watch_flush enter 2024-03-29T10:55:45.819 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.838+0000 7f99fcff9640 10 librados: watch_flush exit 2024-03-29T10:55:45.819 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.838+0000 7f99fcff9640 20 client.4383.objecter shutdown clearing up homeless session... 2024-03-29T10:55:45.819 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.838+0000 7f99fcff9640 10 client.4383.objecter successfully canceled tick 2024-03-29T10:55:45.819 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.838+0000 7f99fcff9640 1 -- 172.21.15.112:0/4243573595 >> v1:172.21.15.118:6816/3181073408 conn(0x7f99ec06dbd0 legacy=0x7f99ec070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:45.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.838+0000 7f99fcff9640 10 monclient: shutdown 2024-03-29T10:55:45.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.838+0000 7f99fcff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:45.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.838+0000 7f99fcff9640 1 -- 172.21.15.112:0/4243573595 >> v1:172.21.15.112:6789/0 conn(0x7f9a080a9d70 legacy=0x7f9a081394d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:45.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.838+0000 7f99fcff9640 1 -- 172.21.15.112:0/4243573595 shutdown_connections 2024-03-29T10:55:45.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.838+0000 7f99fcff9640 1 -- 172.21.15.112:0/4243573595 >> 172.21.15.112:0/4243573595 conn(0x7f9a08066a00 msgr2=0x7f9a080b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:45.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.838+0000 7f99fcff9640 1 -- 172.21.15.112:0/4243573595 shutdown_connections 2024-03-29T10:55:45.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.838+0000 7f99fcff9640 1 -- 172.21.15.112:0/4243573595 wait complete. 2024-03-29T10:55:45.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:45.838+0000 7f99fcff9640 1 librados: shutdown 2024-03-29T10:55:45.832 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:52.639 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:52.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.793+0000 7f9280e0f640 10 monclient: get_monmap_and_config 2024-03-29T10:55:52.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.793+0000 7f9280e0f640 10 monclient: build_initial_monmap 2024-03-29T10:55:52.775 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.793+0000 7f9280e0f640 10 monclient: monmap: 2024-03-29T10:55:52.775 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:55:52.775 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:52.775 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:55:52.794726+0000 2024-03-29T10:55:52.775 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:55:52.794726+0000 2024-03-29T10:55:52.775 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:52.775 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:52.776 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:55:52.776 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:55:52.776 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:55:52.776 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:52.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.794+0000 7f9280e0f640 1 Processor -- start 2024-03-29T10:55:52.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.794+0000 7f9280e0f640 1 -- start start 2024-03-29T10:55:52.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.794+0000 7f9280e0f640 10 monclient: init 2024-03-29T10:55:52.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.795+0000 7f9280e0f640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:52.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.795+0000 7f9280e0f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:55:52.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.795+0000 7f9280e0f640 10 monclient(hunting): picked mon.noname-a con 0x7f927c1422b0 addr v1:172.21.15.112:6789/0 2024-03-29T10:55:52.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.795+0000 7f9280e0f640 10 monclient(hunting): picked mon.noname-b con 0x7f927c142690 addr v1:172.21.15.112:6790/0 2024-03-29T10:55:52.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.795+0000 7f9280e0f640 10 monclient(hunting): picked mon.noname-c con 0x7f927c0b17d0 addr v1:172.21.15.118:6789/0 2024-03-29T10:55:52.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.795+0000 7f9280e0f640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f927c142c80 con 0x7f927c1422b0 2024-03-29T10:55:52.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.795+0000 7f9280e0f640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f927c0aae40 con 0x7f927c0b17d0 2024-03-29T10:55:52.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.795+0000 7f9280e0f640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f927c13fbd0 con 0x7f927c142690 2024-03-29T10:55:52.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.795+0000 7f9280e0f640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:52.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.795+0000 7f9280e0f640 10 monclient(hunting): authenticate will time out at 1108.099668s 2024-03-29T10:55:52.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.795+0000 7f9279d9b640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f927c142690 0x7f927c0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:55816/0 (socket says 172.21.15.112:55816) 2024-03-29T10:55:52.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.795+0000 7f9279d9b640 1 -- 172.21.15.112:0/131803350 learned_addr learned my addr 172.21.15.112:0/131803350 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:55:52.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.796+0000 7f927959a640 1 -- 172.21.15.112:0/131803350 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392079415 0 0) 0x7f927c13fbd0 con 0x7f927c142690 2024-03-29T10:55:52.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.796+0000 7f927959a640 10 monclient(hunting): _negotiate 2024-03-29T10:55:52.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.796+0000 7f927959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:52.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.796+0000 7f927959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:52.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.796+0000 7f927959a640 10 monclient(hunting): my global_id is 4394 2024-03-29T10:55:52.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.796+0000 7f927959a640 1 -- 172.21.15.112:0/131803350 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9264003520 con 0x7f927c142690 2024-03-29T10:55:52.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.796+0000 7f927959a640 1 -- 172.21.15.112:0/131803350 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940069752 0 0) 0x7f927c0aae40 con 0x7f927c0b17d0 2024-03-29T10:55:52.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.796+0000 7f927959a640 10 monclient(hunting): _negotiate 2024-03-29T10:55:52.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.796+0000 7f927959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:52.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.796+0000 7f927959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:52.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.796+0000 7f927959a640 10 monclient(hunting): my global_id is 4378 2024-03-29T10:55:52.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.796+0000 7f927959a640 1 -- 172.21.15.112:0/131803350 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f927c13fbd0 con 0x7f927c0b17d0 2024-03-29T10:55:52.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.796+0000 7f927959a640 1 -- 172.21.15.112:0/131803350 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 760255159 0 0) 0x7f9264003520 con 0x7f927c142690 2024-03-29T10:55:52.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.796+0000 7f927959a640 1 -- 172.21.15.112:0/131803350 >> v1:172.21.15.118:6789/0 conn(0x7f927c0b17d0 legacy=0x7f927c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:52.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.796+0000 7f927959a640 1 -- 172.21.15.112:0/131803350 >> v1:172.21.15.112:6789/0 conn(0x7f927c1422b0 legacy=0x7f927c13f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:52.782 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.798+0000 7f927959a640 10 monclient: _finish_hunting 0 2024-03-29T10:55:52.782 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.798+0000 7f927959a640 1 monclient: found mon.noname-b 2024-03-29T10:55:52.782 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.798+0000 7f927959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:52.782 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.798+0000 7f927959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:55:52.782 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.798+0000 7f927959a640 1 -- 172.21.15.112:0/131803350 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927c151940 con 0x7f927c142690 2024-03-29T10:55:52.782 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.798+0000 7f927959a640 10 monclient: _finish_auth 0 2024-03-29T10:55:52.782 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.798+0000 7f927959a640 10 monclient: _check_auth_tickets 2024-03-29T10:55:52.782 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.798+0000 7f927959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:52.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.798+0000 7f927959a640 1 -- 172.21.15.112:0/131803350 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f926c001690 con 0x7f927c142690 2024-03-29T10:55:52.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.798+0000 7f927959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:52.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.798+0000 7f927959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:52.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.798+0000 7f927959a640 10 monclient: dump: 2024-03-29T10:55:52.783 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:52.784 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:52.784 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:52.784 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:52.784 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:52.784 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:52.784 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:52.784 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:52.784 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:52.785 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:52.785 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.798+0000 7f9280e0f640 5 monclient: authenticate success, global_id 4394 2024-03-29T10:55:52.785 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.798+0000 7f9280e0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:52.785 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.799+0000 7f927959a640 1 -- 172.21.15.112:0/131803350 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f926c0017f0 con 0x7f927c142690 2024-03-29T10:55:52.785 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.799+0000 7f927959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:52.785 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.799+0000 7f927959a640 1 -- 172.21.15.112:0/131803350 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f926c001b80 con 0x7f927c142690 2024-03-29T10:55:52.785 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.799+0000 7f927959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:52.785 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.799+0000 7f927959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:52.785 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.799+0000 7f927959a640 10 monclient: dump: 2024-03-29T10:55:52.785 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:52.785 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:52.785 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:52.785 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:52.785 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:52.786 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:52.786 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:52.786 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:52.786 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:52.786 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:52.786 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.799+0000 7f9280e0f640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:52.786 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.799+0000 7f9280e0f640 4 set_mon_vals no callback set 2024-03-29T10:55:52.786 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.800+0000 7f9280e0f640 10 monclient: shutdown 2024-03-29T10:55:52.786 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.800+0000 7f9280e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:52.786 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.800+0000 7f9280e0f640 1 -- 172.21.15.112:0/131803350 >> v1:172.21.15.112:6790/0 conn(0x7f927c142690 legacy=0x7f927c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:52.786 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.800+0000 7f9280e0f640 1 -- 172.21.15.112:0/131803350 shutdown_connections 2024-03-29T10:55:52.786 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.800+0000 7f9280e0f640 1 -- 172.21.15.112:0/131803350 >> 172.21.15.112:0/131803350 conn(0x7f927c066a00 msgr2=0x7f927c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:52.787 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.800+0000 7f9280e0f640 1 -- 172.21.15.112:0/131803350 shutdown_connections 2024-03-29T10:55:52.787 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.801+0000 7f9280e0f640 1 -- 172.21.15.112:0/131803350 wait complete. 2024-03-29T10:55:52.793 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.802+0000 7f9280e0f640 10 monclient: build_initial_monmap 2024-03-29T10:55:52.793 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.802+0000 7f9280e0f640 10 monclient: monmap: 2024-03-29T10:55:52.793 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:55:52.793 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:52.794 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:55:52.794 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:55:52.794 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:52.794 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:52.794 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:55:52.794 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:55:52.794 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:55:52.794 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:52.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.802+0000 7f9280e0f640 1 librados: starting msgr at 2024-03-29T10:55:52.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.802+0000 7f9280e0f640 1 librados: starting objecter 2024-03-29T10:55:52.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.802+0000 7f9280e0f640 1 Processor -- start 2024-03-29T10:55:52.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.802+0000 7f9280e0f640 1 -- start start 2024-03-29T10:55:52.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.802+0000 7f9280e0f640 1 librados: setting wanted keys 2024-03-29T10:55:52.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.802+0000 7f9280e0f640 1 librados: calling monclient init 2024-03-29T10:55:52.795 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.802+0000 7f9280e0f640 10 monclient: init 2024-03-29T10:55:52.795 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.803+0000 7f9280e0f640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:52.795 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.803+0000 7f9280e0f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:55:52.795 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.803+0000 7f9280e0f640 10 monclient(hunting): picked mon.noname-a con 0x7f927c0b17d0 addr v1:172.21.15.112:6789/0 2024-03-29T10:55:52.795 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.803+0000 7f9280e0f640 10 monclient(hunting): picked mon.noname-c con 0x7f927c1422b0 addr v1:172.21.15.112:6790/0 2024-03-29T10:55:52.795 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.803+0000 7f9280e0f640 10 monclient(hunting): picked mon.noname-b con 0x7f927c142690 addr v1:172.21.15.118:6789/0 2024-03-29T10:55:52.795 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.803+0000 7f9280e0f640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f927c131920 con 0x7f927c0b17d0 2024-03-29T10:55:52.795 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.803+0000 7f9280e0f640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f927c0abc10 con 0x7f927c142690 2024-03-29T10:55:52.795 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.803+0000 7f9280e0f640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f927c0ac160 con 0x7f927c1422b0 2024-03-29T10:55:52.795 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.803+0000 7f9280e0f640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:52.795 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.803+0000 7f9280e0f640 10 monclient(hunting): authenticate will time out at 1108.108080s 2024-03-29T10:55:52.795 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.803+0000 7f9279d9b640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f927c1422b0 0x7f927c1324c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:58478/0 (socket says 172.21.15.112:58478) 2024-03-29T10:55:52.795 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.803+0000 7f9279d9b640 1 -- 172.21.15.112:0/2024404568 learned_addr learned my addr 172.21.15.112:0/2024404568 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:55:52.796 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.804+0000 7f926affd640 10 client.?.objecter ms_handle_connect 0x7f927c1422b0 2024-03-29T10:55:52.796 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.804+0000 7f926affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:52.796 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.804+0000 7f926affd640 1 -- 172.21.15.112:0/2024404568 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419032488 0 0) 0x7f927c0ac160 con 0x7f927c1422b0 2024-03-29T10:55:52.796 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.804+0000 7f926affd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:52.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.804+0000 7f926affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:52.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.804+0000 7f926affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:52.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.804+0000 7f926affd640 10 monclient(hunting): my global_id is 4397 2024-03-29T10:55:52.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.804+0000 7f926affd640 1 -- 172.21.15.112:0/2024404568 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f925c004eb0 con 0x7f927c1422b0 2024-03-29T10:55:52.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.804+0000 7f926affd640 1 -- 172.21.15.112:0/2024404568 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3354680006 0 0) 0x7f925c004eb0 con 0x7f927c1422b0 2024-03-29T10:55:52.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.804+0000 7f926affd640 1 -- 172.21.15.112:0/2024404568 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f927c0ac160 con 0x7f927c1422b0 2024-03-29T10:55:52.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.804+0000 7f926affd640 1 -- 172.21.15.112:0/2024404568 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f926c001bc0 con 0x7f927c1422b0 2024-03-29T10:55:52.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.804+0000 7f926affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:52.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.804+0000 7f926affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:52.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.804+0000 7f926affd640 10 monclient(hunting): dump: 2024-03-29T10:55:52.798 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:52.798 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:52.798 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:52.798 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:52.798 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:52.798 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:52.798 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:52.798 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:52.798 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:52.799 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:52.799 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.804+0000 7f926affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:52.799 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.804+0000 7f926affd640 10 client.?.objecter ms_handle_connect 0x7f927c142690 2024-03-29T10:55:52.799 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.804+0000 7f926affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:52.799 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.805+0000 7f926affd640 1 -- 172.21.15.112:0/2024404568 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2044433377 0 0) 0x7f927c0ac160 con 0x7f927c1422b0 2024-03-29T10:55:52.799 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.805+0000 7f926affd640 1 -- 172.21.15.112:0/2024404568 >> v1:172.21.15.118:6789/0 conn(0x7f927c142690 legacy=0x7f927c1e1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:52.799 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.805+0000 7f926affd640 1 -- 172.21.15.112:0/2024404568 >> v1:172.21.15.112:6789/0 conn(0x7f927c0b17d0 legacy=0x7f927c131db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:52.799 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.806+0000 7f926affd640 10 monclient: _finish_hunting 0 2024-03-29T10:55:52.799 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.806+0000 7f926affd640 1 monclient: found mon.c 2024-03-29T10:55:52.799 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.806+0000 7f926affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:52.799 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.806+0000 7f926affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:55:52.799 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.806+0000 7f926affd640 1 -- 172.21.15.112:0/2024404568 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927c0abe80 con 0x7f927c1422b0 2024-03-29T10:55:52.799 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.806+0000 7f926affd640 10 monclient: _finish_auth 0 2024-03-29T10:55:52.799 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.806+0000 7f926affd640 10 monclient: _check_auth_tickets 2024-03-29T10:55:52.799 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.806+0000 7f926affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:52.800 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.806+0000 7f9280e0f640 5 monclient: authenticate success, global_id 4397 2024-03-29T10:55:52.800 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.806+0000 7f9280e0f640 10 monclient: _renew_subs 2024-03-29T10:55:52.800 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.806+0000 7f9280e0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:55:52.800 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.806+0000 7f9280e0f640 1 -- 172.21.15.112:0/2024404568 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f927c0b08d0 con 0x7f927c1422b0 2024-03-29T10:55:52.800 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.806+0000 7f9280e0f640 10 client.4397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:52.800 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.806+0000 7f9280e0f640 10 monclient: _renew_subs 2024-03-29T10:55:52.800 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.806+0000 7f9280e0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:55:52.800 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.807+0000 7f9280e0f640 1 -- 172.21.15.112:0/2024404568 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f927c150a40 con 0x7f927c1422b0 2024-03-29T10:55:52.800 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.807+0000 7f9280e0f640 1 librados: init done 2024-03-29T10:55:52.800 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.807+0000 7f926affd640 1 -- 172.21.15.112:0/2024404568 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f926c001270 con 0x7f927c1422b0 2024-03-29T10:55:52.800 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.807+0000 7f926affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:52.800 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.807+0000 7f9278d99640 4 set_mon_vals no callback set 2024-03-29T10:55:52.800 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.807+0000 7f926affd640 1 -- 172.21.15.112:0/2024404568 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f926c004c30 con 0x7f927c1422b0 2024-03-29T10:55:52.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.807+0000 7f926affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:52.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.807+0000 7f926affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:52.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.807+0000 7f926affd640 10 monclient: dump: 2024-03-29T10:55:52.801 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:55:52.801 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:55:52.801 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:52.801 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:55:52.801 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:52.801 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:55:52.801 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:55:52.801 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:55:52.801 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:55:52.801 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:55:52.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.808+0000 7f9280e0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:52.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.808+0000 7f9280e0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:52.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.808+0000 7f9280e0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:55:52.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.808+0000 7f9280e0f640 1 -- 172.21.15.112:0/2024404568 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9260005080 con 0x7f927c1422b0 2024-03-29T10:55:52.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.809+0000 7f926affd640 1 -- 172.21.15.112:0/2024404568 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f926c01ac00 con 0x7f927c1422b0 2024-03-29T10:55:52.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.810+0000 7f926affd640 1 -- 172.21.15.112:0/2024404568 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f926c084bd0 con 0x7f927c1422b0 2024-03-29T10:55:52.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.810+0000 7f926affd640 10 client.4397.objecter ms_dispatch 0x7f927c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:52.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.810+0000 7f926affd640 3 client.4397.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:52.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.810+0000 7f926affd640 3 client.4397.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:52.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.810+0000 7f927a59c640 10 monclient: get_auth_request con 0x7f925c06dbd0 auth_method 0 2024-03-29T10:55:52.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.810+0000 7f926affd640 20 client.4397.objecter dump_active .. 0 homeless 2024-03-29T10:55:52.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.810+0000 7f926affd640 10 client.4397.objecter ms_handle_connect 0x7f925c06dbd0 2024-03-29T10:55:52.803 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.816+0000 7f926affd640 1 -- 172.21.15.112:0/2024404568 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f926c0502e0 con 0x7f927c1422b0 2024-03-29T10:55:52.803 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.816+0000 7f926affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:52.803 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:52.816+0000 7f926affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:53.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:53.114+0000 7f9280e0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:53.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:53.114+0000 7f9280e0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:53.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:53.114+0000 7f9280e0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:55:53.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:53.114+0000 7f9280e0f640 1 -- 172.21.15.112:0/2024404568 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9260005330 con 0x7f927c1422b0 2024-03-29T10:55:53.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:53.116+0000 7f926affd640 1 -- 172.21.15.112:0/2024404568 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f926c0523e0 con 0x7f927c1422b0 2024-03-29T10:55:53.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:53.116+0000 7f926affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:53.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:53.116+0000 7f926affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:53.097 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:55:53.097 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:53.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:53.119+0000 7f9280e0f640 10 librados: watch_flush enter 2024-03-29T10:55:53.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:53.119+0000 7f9280e0f640 10 librados: watch_flush exit 2024-03-29T10:55:53.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:53.119+0000 7f9280e0f640 20 client.4397.objecter shutdown clearing up homeless session... 2024-03-29T10:55:53.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:53.119+0000 7f9280e0f640 10 client.4397.objecter successfully canceled tick 2024-03-29T10:55:53.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:53.119+0000 7f9280e0f640 1 -- 172.21.15.112:0/2024404568 >> v1:172.21.15.118:6816/3181073408 conn(0x7f925c06dbd0 legacy=0x7f925c070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:53.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:53.119+0000 7f9280e0f640 10 monclient: shutdown 2024-03-29T10:55:53.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:53.119+0000 7f9280e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:53.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:53.119+0000 7f9280e0f640 1 -- 172.21.15.112:0/2024404568 >> v1:172.21.15.112:6790/0 conn(0x7f927c1422b0 legacy=0x7f927c1324c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:53.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:53.119+0000 7f9280e0f640 1 -- 172.21.15.112:0/2024404568 shutdown_connections 2024-03-29T10:55:53.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:53.119+0000 7f9280e0f640 1 -- 172.21.15.112:0/2024404568 >> 172.21.15.112:0/2024404568 conn(0x7f927c066a00 msgr2=0x7f927c13c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:53.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:53.120+0000 7f9280e0f640 1 -- 172.21.15.112:0/2024404568 shutdown_connections 2024-03-29T10:55:53.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:53.120+0000 7f9280e0f640 1 -- 172.21.15.112:0/2024404568 wait complete. 2024-03-29T10:55:53.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:55:53.120+0000 7f9280e0f640 1 librados: shutdown 2024-03-29T10:55:53.114 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:59.921 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:00.071 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.089+0000 7f5ead427640 10 monclient: get_monmap_and_config 2024-03-29T10:56:00.071 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.089+0000 7f5ead427640 10 monclient: build_initial_monmap 2024-03-29T10:56:00.071 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.089+0000 7f5ead427640 10 monclient: monmap: 2024-03-29T10:56:00.071 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:56:00.071 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:00.071 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:56:00.091455+0000 2024-03-29T10:56:00.071 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:56:00.091455+0000 2024-03-29T10:56:00.072 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:00.072 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:00.072 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:56:00.072 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:56:00.072 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:56:00.072 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:00.072 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.090+0000 7f5ead427640 1 Processor -- start 2024-03-29T10:56:00.073 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.090+0000 7f5ead427640 1 -- start start 2024-03-29T10:56:00.073 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.090+0000 7f5ead427640 10 monclient: init 2024-03-29T10:56:00.073 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.091+0000 7f5ead427640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:00.073 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.091+0000 7f5ead427640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:56:00.076 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.091+0000 7f5ead427640 10 monclient(hunting): picked mon.noname-b con 0x7f5ea80b1f60 addr v1:172.21.15.112:6790/0 2024-03-29T10:56:00.076 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.091+0000 7f5ead427640 10 monclient(hunting): picked mon.noname-a con 0x7f5ea8071eb0 addr v1:172.21.15.112:6789/0 2024-03-29T10:56:00.076 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.091+0000 7f5ead427640 10 monclient(hunting): picked mon.noname-c con 0x7f5ea8143d10 addr v1:172.21.15.118:6789/0 2024-03-29T10:56:00.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.091+0000 7f5ead427640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea806ef80 con 0x7f5ea8071eb0 2024-03-29T10:56:00.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.091+0000 7f5ead427640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea806f260 con 0x7f5ea8143d10 2024-03-29T10:56:00.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.094+0000 7f5ea6ffd640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f5ea80b1f60 0x7f5ea8143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35038/0 (socket says 172.21.15.112:35038) 2024-03-29T10:56:00.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.094+0000 7f5ea6ffd640 1 -- 172.21.15.112:0/3547781757 learned_addr learned my addr 172.21.15.112:0/3547781757 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:56:00.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.094+0000 7f5ead427640 1 -- 172.21.15.112:0/3547781757 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea80b40c0 con 0x7f5ea80b1f60 2024-03-29T10:56:00.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.094+0000 7f5ead427640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:00.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.094+0000 7f5ead427640 10 monclient(hunting): authenticate will time out at 1115.399938s 2024-03-29T10:56:00.080 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.095+0000 7f5ea5ffb640 1 -- 172.21.15.112:0/3547781757 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082243464 0 0) 0x7f5ea80b40c0 con 0x7f5ea80b1f60 2024-03-29T10:56:00.080 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.095+0000 7f5ea5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:56:00.080 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.095+0000 7f5ea5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:00.080 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.095+0000 7f5ea5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:00.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.095+0000 7f5ea5ffb640 10 monclient(hunting): my global_id is 4400 2024-03-29T10:56:00.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 1 -- 172.21.15.112:0/3547781757 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e90003520 con 0x7f5ea80b1f60 2024-03-29T10:56:00.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 1 -- 172.21.15.112:0/3547781757 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551712184 0 0) 0x7f5ea806ef80 con 0x7f5ea8071eb0 2024-03-29T10:56:00.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:56:00.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:00.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:00.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 10 monclient(hunting): my global_id is 4386 2024-03-29T10:56:00.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 1 -- 172.21.15.112:0/3547781757 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ea80b40c0 con 0x7f5ea8071eb0 2024-03-29T10:56:00.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 1 -- 172.21.15.112:0/3547781757 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2906272008 0 0) 0x7f5e90003520 con 0x7f5ea80b1f60 2024-03-29T10:56:00.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 1 -- 172.21.15.112:0/3547781757 >> v1:172.21.15.118:6789/0 conn(0x7f5ea8143d10 legacy=0x7f5ea8146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:56:00.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 1 -- 172.21.15.112:0/3547781757 >> v1:172.21.15.112:6789/0 conn(0x7f5ea8071eb0 legacy=0x7f5ea8146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:00.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:56:00.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 1 monclient: found mon.noname-b 2024-03-29T10:56:00.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:00.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:56:00.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 1 -- 172.21.15.112:0/3547781757 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea8155d70 con 0x7f5ea80b1f60 2024-03-29T10:56:00.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:56:00.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:56:00.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:00.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 1 -- 172.21.15.112:0/3547781757 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f5e98001690 con 0x7f5ea80b1f60 2024-03-29T10:56:00.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:00.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:00.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 10 monclient: dump: 2024-03-29T10:56:00.082 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:00.082 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:00.082 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:00.082 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:00.083 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:00.083 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:00.083 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:00.083 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:00.083 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:00.083 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:00.083 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 1 -- 172.21.15.112:0/3547781757 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e980017f0 con 0x7f5ea80b1f60 2024-03-29T10:56:00.083 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:00.083 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 1 -- 172.21.15.112:0/3547781757 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f5e98001c00 con 0x7f5ea80b1f60 2024-03-29T10:56:00.083 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:00.083 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:00.083 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.096+0000 7f5ea5ffb640 10 monclient: dump: 2024-03-29T10:56:00.083 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:00.083 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:00.084 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:00.084 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:00.084 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:00.084 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:00.084 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:00.084 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:00.084 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:00.084 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:00.084 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.097+0000 7f5ead427640 5 monclient: authenticate success, global_id 4400 2024-03-29T10:56:00.084 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.097+0000 7f5ead427640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:00.084 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.097+0000 7f5ead427640 4 set_mon_vals no callback set 2024-03-29T10:56:00.084 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.098+0000 7f5ead427640 10 monclient: shutdown 2024-03-29T10:56:00.084 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.098+0000 7f5ead427640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:00.084 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.098+0000 7f5ead427640 1 -- 172.21.15.112:0/3547781757 >> v1:172.21.15.112:6790/0 conn(0x7f5ea80b1f60 legacy=0x7f5ea8143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:00.088 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.099+0000 7f5ead427640 1 -- 172.21.15.112:0/3547781757 shutdown_connections 2024-03-29T10:56:00.088 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.099+0000 7f5ead427640 1 -- 172.21.15.112:0/3547781757 >> 172.21.15.112:0/3547781757 conn(0x7f5ea8066a00 msgr2=0x7f5ea8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:00.088 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.101+0000 7f5ead427640 1 -- 172.21.15.112:0/3547781757 shutdown_connections 2024-03-29T10:56:00.088 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.101+0000 7f5ead427640 1 -- 172.21.15.112:0/3547781757 wait complete. 2024-03-29T10:56:00.088 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.102+0000 7f5ead427640 10 monclient: build_initial_monmap 2024-03-29T10:56:00.088 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.102+0000 7f5ead427640 10 monclient: monmap: 2024-03-29T10:56:00.088 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:56:00.088 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:00.088 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:56:00.088 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:56:00.089 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:00.089 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:00.089 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:56:00.089 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:56:00.089 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:56:00.089 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:00.089 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.102+0000 7f5ead427640 1 librados: starting msgr at 2024-03-29T10:56:00.089 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.102+0000 7f5ead427640 1 librados: starting objecter 2024-03-29T10:56:00.089 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.103+0000 7f5ead427640 1 Processor -- start 2024-03-29T10:56:00.089 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.103+0000 7f5ead427640 1 -- start start 2024-03-29T10:56:00.089 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.103+0000 7f5ead427640 1 librados: setting wanted keys 2024-03-29T10:56:00.089 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.103+0000 7f5ead427640 1 librados: calling monclient init 2024-03-29T10:56:00.089 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.103+0000 7f5ead427640 10 monclient: init 2024-03-29T10:56:00.089 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.104+0000 7f5ead427640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:00.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.104+0000 7f5ead427640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:00.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.104+0000 7f5ead427640 10 monclient(hunting): picked mon.noname-c con 0x7f5ea80b1f60 addr v1:172.21.15.112:6790/0 2024-03-29T10:56:00.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.104+0000 7f5ead427640 10 monclient(hunting): picked mon.noname-b con 0x7f5ea8143d10 addr v1:172.21.15.118:6789/0 2024-03-29T10:56:00.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.104+0000 7f5ead427640 10 monclient(hunting): picked mon.noname-a con 0x7f5ea8147790 addr v1:172.21.15.112:6789/0 2024-03-29T10:56:00.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.104+0000 7f5ead427640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea814ab40 con 0x7f5ea8147790 2024-03-29T10:56:00.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.104+0000 7f5ead427640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea8147cb0 con 0x7f5ea8143d10 2024-03-29T10:56:00.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.104+0000 7f5ead427640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea81dfa70 con 0x7f5ea80b1f60 2024-03-29T10:56:00.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.104+0000 7f5ead427640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:00.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.104+0000 7f5ead427640 10 monclient(hunting): authenticate will time out at 1115.409720s 2024-03-29T10:56:00.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.104+0000 7f5ea6ffd640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f5ea80b1f60 0x7f5ea813f390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:39780/0 (socket says 172.21.15.112:39780) 2024-03-29T10:56:00.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.104+0000 7f5ea6ffd640 1 -- 172.21.15.112:0/2810691268 learned_addr learned my addr 172.21.15.112:0/2810691268 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:56:00.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.104+0000 7f5e8f7fe640 10 client.?.objecter ms_handle_connect 0x7f5ea80b1f60 2024-03-29T10:56:00.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.104+0000 7f5e8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:00.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.105+0000 7f5e8f7fe640 1 -- 172.21.15.112:0/2810691268 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 467322006 0 0) 0x7f5ea81dfa70 con 0x7f5ea80b1f60 2024-03-29T10:56:00.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.105+0000 7f5e8f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:56:00.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.105+0000 7f5e8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:00.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.105+0000 7f5e8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:00.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.105+0000 7f5e8f7fe640 10 monclient(hunting): my global_id is 4403 2024-03-29T10:56:00.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.105+0000 7f5e8f7fe640 1 -- 172.21.15.112:0/2810691268 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e94004eb0 con 0x7f5ea80b1f60 2024-03-29T10:56:00.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.105+0000 7f5e8f7fe640 1 -- 172.21.15.112:0/2810691268 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2974754046 0 0) 0x7f5e94004eb0 con 0x7f5ea80b1f60 2024-03-29T10:56:00.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.105+0000 7f5e8f7fe640 1 -- 172.21.15.112:0/2810691268 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ea81dfa70 con 0x7f5ea80b1f60 2024-03-29T10:56:00.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.105+0000 7f5e8f7fe640 1 -- 172.21.15.112:0/2810691268 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f5e98001c00 con 0x7f5ea80b1f60 2024-03-29T10:56:00.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.105+0000 7f5e8f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:00.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.105+0000 7f5e8f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:00.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.105+0000 7f5e8f7fe640 10 monclient(hunting): dump: 2024-03-29T10:56:00.091 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:00.092 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:00.092 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:00.092 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:00.092 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:00.092 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:00.092 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:00.092 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:00.092 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:00.092 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:00.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.105+0000 7f5e8f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:00.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.106+0000 7f5e8f7fe640 1 -- 172.21.15.112:0/2810691268 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4083081037 0 0) 0x7f5ea81dfa70 con 0x7f5ea80b1f60 2024-03-29T10:56:00.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.106+0000 7f5e8f7fe640 1 -- 172.21.15.112:0/2810691268 >> v1:172.21.15.118:6789/0 conn(0x7f5ea8143d10 legacy=0x7f5ea81dec50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:00.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.106+0000 7f5e8f7fe640 1 -- 172.21.15.112:0/2810691268 >> v1:172.21.15.112:6789/0 conn(0x7f5ea8147790 legacy=0x7f5ea81df360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:00.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.108+0000 7f5e8f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:56:00.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.108+0000 7f5e8f7fe640 1 monclient: found mon.c 2024-03-29T10:56:00.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.108+0000 7f5e8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:00.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.108+0000 7f5e8f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:00.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.108+0000 7f5e8f7fe640 1 -- 172.21.15.112:0/2810691268 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea814ad70 con 0x7f5ea80b1f60 2024-03-29T10:56:00.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.108+0000 7f5e8f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:56:00.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.108+0000 7f5e8f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:56:00.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.108+0000 7f5e8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:00.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.108+0000 7f5ead427640 5 monclient: authenticate success, global_id 4403 2024-03-29T10:56:00.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.108+0000 7f5ead427640 10 monclient: _renew_subs 2024-03-29T10:56:00.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.109+0000 7f5ead427640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:00.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.109+0000 7f5ead427640 1 -- 172.21.15.112:0/2810691268 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea814b050 con 0x7f5ea80b1f60 2024-03-29T10:56:00.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.109+0000 7f5ead427640 10 client.4403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:00.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.109+0000 7f5ead427640 10 monclient: _renew_subs 2024-03-29T10:56:00.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.109+0000 7f5ead427640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:00.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.109+0000 7f5ead427640 1 -- 172.21.15.112:0/2810691268 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea80708e0 con 0x7f5ea80b1f60 2024-03-29T10:56:00.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.109+0000 7f5ead427640 1 librados: init done 2024-03-29T10:56:00.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.110+0000 7f5e8f7fe640 1 -- 172.21.15.112:0/2810691268 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e98004b50 con 0x7f5ea80b1f60 2024-03-29T10:56:00.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.110+0000 7f5e8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:00.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.110+0000 7f5e8f7fe640 1 -- 172.21.15.112:0/2810691268 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f5e98004f20 con 0x7f5ea80b1f60 2024-03-29T10:56:00.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.110+0000 7f5e8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:00.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.110+0000 7f5e8f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:00.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.110+0000 7f5e8f7fe640 10 monclient: dump: 2024-03-29T10:56:00.097 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:00.097 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:00.097 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:00.097 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:00.097 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:00.097 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:00.097 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:00.097 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:00.098 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:00.098 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:00.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.110+0000 7f5ea57fa640 4 set_mon_vals no callback set 2024-03-29T10:56:00.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.110+0000 7f5ead427640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:00.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.111+0000 7f5ead427640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:00.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.111+0000 7f5ead427640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:00.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.111+0000 7f5ead427640 1 -- 172.21.15.112:0/2810691268 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e88005080 con 0x7f5ea80b1f60 2024-03-29T10:56:00.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.112+0000 7f5e8f7fe640 1 -- 172.21.15.112:0/2810691268 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f5e98005080 con 0x7f5ea80b1f60 2024-03-29T10:56:00.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.113+0000 7f5e8f7fe640 1 -- 172.21.15.112:0/2810691268 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f5e98084d90 con 0x7f5ea80b1f60 2024-03-29T10:56:00.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.113+0000 7f5e8f7fe640 10 client.4403.objecter ms_dispatch 0x7f5ea8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:00.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.113+0000 7f5e8f7fe640 3 client.4403.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:00.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.113+0000 7f5e8f7fe640 3 client.4403.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:00.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.113+0000 7f5e8f7fe640 20 client.4403.objecter dump_active .. 0 homeless 2024-03-29T10:56:00.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.114+0000 7f5ea67fc640 10 monclient: get_auth_request con 0x7f5e9406dc80 auth_method 0 2024-03-29T10:56:00.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.115+0000 7f5e8f7fe640 10 client.4403.objecter ms_handle_connect 0x7f5e9406dc80 2024-03-29T10:56:00.102 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.121+0000 7f5e8f7fe640 1 -- 172.21.15.112:0/2810691268 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5e98050670 con 0x7f5ea80b1f60 2024-03-29T10:56:00.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.121+0000 7f5e8f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:00.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.121+0000 7f5e8f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:00.420 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.438+0000 7f5ead427640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:00.421 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.438+0000 7f5ead427640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:00.421 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.438+0000 7f5ead427640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:00.421 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.439+0000 7f5ead427640 1 -- 172.21.15.112:0/2810691268 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e88005330 con 0x7f5ea80b1f60 2024-03-29T10:56:00.423 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.441+0000 7f5e8f7fe640 1 -- 172.21.15.112:0/2810691268 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f5e980525a0 con 0x7f5ea80b1f60 2024-03-29T10:56:00.423 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.441+0000 7f5e8f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:00.423 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.441+0000 7f5e8f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:00.423 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:56:00.423 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:00.426 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.444+0000 7f5ead427640 10 librados: watch_flush enter 2024-03-29T10:56:00.426 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.444+0000 7f5ead427640 10 librados: watch_flush exit 2024-03-29T10:56:00.426 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.444+0000 7f5ead427640 20 client.4403.objecter shutdown clearing up homeless session... 2024-03-29T10:56:00.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.444+0000 7f5ead427640 10 client.4403.objecter successfully canceled tick 2024-03-29T10:56:00.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.445+0000 7f5ead427640 1 -- 172.21.15.112:0/2810691268 >> v1:172.21.15.118:6816/3181073408 conn(0x7f5e9406dc80 legacy=0x7f5e94070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:00.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.445+0000 7f5ead427640 10 monclient: shutdown 2024-03-29T10:56:00.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.445+0000 7f5ead427640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:00.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.445+0000 7f5ead427640 1 -- 172.21.15.112:0/2810691268 >> v1:172.21.15.112:6790/0 conn(0x7f5ea80b1f60 legacy=0x7f5ea813f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:00.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.447+0000 7f5ead427640 1 -- 172.21.15.112:0/2810691268 shutdown_connections 2024-03-29T10:56:00.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.447+0000 7f5ead427640 1 -- 172.21.15.112:0/2810691268 >> 172.21.15.112:0/2810691268 conn(0x7f5ea8066a00 msgr2=0x7f5ea80b7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:00.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.447+0000 7f5ead427640 1 -- 172.21.15.112:0/2810691268 shutdown_connections 2024-03-29T10:56:00.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.447+0000 7f5ead427640 1 -- 172.21.15.112:0/2810691268 wait complete. 2024-03-29T10:56:00.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:00.447+0000 7f5ead427640 1 librados: shutdown 2024-03-29T10:56:00.446 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:07.253 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:07.401 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.420+0000 7f2615d15640 10 monclient: get_monmap_and_config 2024-03-29T10:56:07.402 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.420+0000 7f2615d15640 10 monclient: build_initial_monmap 2024-03-29T10:56:07.402 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.420+0000 7f2615d15640 10 monclient: monmap: 2024-03-29T10:56:07.402 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:56:07.402 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:07.402 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:56:07.421195+0000 2024-03-29T10:56:07.402 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:56:07.421195+0000 2024-03-29T10:56:07.402 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:07.402 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:07.402 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:56:07.402 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:56:07.402 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:56:07.402 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:07.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.421+0000 7f2615d15640 1 Processor -- start 2024-03-29T10:56:07.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.421+0000 7f2615d15640 1 -- start start 2024-03-29T10:56:07.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.421+0000 7f2615d15640 10 monclient: init 2024-03-29T10:56:07.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.422+0000 7f2615d15640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:07.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.422+0000 7f2615d15640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:56:07.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.422+0000 7f2615d15640 10 monclient(hunting): picked mon.noname-c con 0x7f2608034850 addr v1:172.21.15.118:6789/0 2024-03-29T10:56:07.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.423+0000 7f260ffff640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f260802a640 0x7f26080320c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:49444/0 (socket says 172.21.15.112:49444) 2024-03-29T10:56:07.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.423+0000 7f260ffff640 1 -- 172.21.15.112:0/2202156200 learned_addr learned my addr 172.21.15.112:0/2202156200 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:56:07.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.423+0000 7f2615d15640 10 monclient(hunting): picked mon.noname-b con 0x7f260802a640 addr v1:172.21.15.112:6790/0 2024-03-29T10:56:07.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.424+0000 7f2615d15640 10 monclient(hunting): picked mon.noname-a con 0x7f26080327d0 addr v1:172.21.15.112:6789/0 2024-03-29T10:56:07.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.424+0000 7f2615d15640 1 -- 172.21.15.112:0/2202156200 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f260802c820 con 0x7f26080327d0 2024-03-29T10:56:07.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.424+0000 7f2615d15640 1 -- 172.21.15.112:0/2202156200 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f260802a120 con 0x7f2608034850 2024-03-29T10:56:07.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.424+0000 7f2615d15640 1 -- 172.21.15.112:0/2202156200 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26080c75b0 con 0x7f260802a640 2024-03-29T10:56:07.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.424+0000 7f2615d15640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:07.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.424+0000 7f2615d15640 10 monclient(hunting): authenticate will time out at 1122.728743s 2024-03-29T10:56:07.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.425+0000 7f260f7fe640 1 -- 172.21.15.112:0/2202156200 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806621795 0 0) 0x7f26080c75b0 con 0x7f260802a640 2024-03-29T10:56:07.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.425+0000 7f260f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:56:07.407 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.425+0000 7f260f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:07.407 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.425+0000 7f260f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:07.407 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.425+0000 7f260f7fe640 10 monclient(hunting): my global_id is 4406 2024-03-29T10:56:07.407 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.425+0000 7f260f7fe640 1 -- 172.21.15.112:0/2202156200 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2600003520 con 0x7f260802a640 2024-03-29T10:56:07.407 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.425+0000 7f260f7fe640 1 -- 172.21.15.112:0/2202156200 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2286662172 0 0) 0x7f2600003520 con 0x7f260802a640 2024-03-29T10:56:07.407 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.425+0000 7f260f7fe640 1 -- 172.21.15.112:0/2202156200 >> v1:172.21.15.118:6789/0 conn(0x7f2608034850 legacy=0x7f26080c6ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:07.407 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.426+0000 7f260f7fe640 1 -- 172.21.15.112:0/2202156200 >> v1:172.21.15.112:6789/0 conn(0x7f26080327d0 legacy=0x7f26080cf240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:07.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.427+0000 7f260f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:56:07.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.427+0000 7f260f7fe640 1 monclient: found mon.noname-b 2024-03-29T10:56:07.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.427+0000 7f260f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:07.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.427+0000 7f260f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:56:07.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.427+0000 7f260f7fe640 1 -- 172.21.15.112:0/2202156200 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2608033110 con 0x7f260802a640 2024-03-29T10:56:07.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.427+0000 7f260f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:56:07.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.428+0000 7f260f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:56:07.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.428+0000 7f260f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:07.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.428+0000 7f260f7fe640 1 -- 172.21.15.112:0/2202156200 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f2604001690 con 0x7f260802a640 2024-03-29T10:56:07.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.428+0000 7f260f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:07.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.428+0000 7f260f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:07.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.428+0000 7f260f7fe640 10 monclient: dump: 2024-03-29T10:56:07.409 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:07.410 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:07.410 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:07.410 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:07.410 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:07.410 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:07.410 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:07.410 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:07.410 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:07.410 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:07.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.429+0000 7f2615d15640 5 monclient: authenticate success, global_id 4406 2024-03-29T10:56:07.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.429+0000 7f2615d15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:07.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.430+0000 7f260f7fe640 1 -- 172.21.15.112:0/2202156200 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2604001860 con 0x7f260802a640 2024-03-29T10:56:07.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.430+0000 7f260f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:07.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.430+0000 7f260f7fe640 1 -- 172.21.15.112:0/2202156200 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f2604001cb0 con 0x7f260802a640 2024-03-29T10:56:07.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.430+0000 7f260f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:07.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.430+0000 7f260f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:07.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.430+0000 7f260f7fe640 10 monclient: dump: 2024-03-29T10:56:07.412 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:07.412 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:07.412 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:07.412 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:07.412 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:07.412 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:07.412 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:07.412 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:07.412 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:07.412 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:07.413 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.431+0000 7f2615d15640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:07.413 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.431+0000 7f2615d15640 4 set_mon_vals no callback set 2024-03-29T10:56:07.413 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.432+0000 7f2615d15640 10 monclient: shutdown 2024-03-29T10:56:07.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.432+0000 7f2615d15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:07.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.432+0000 7f2615d15640 1 -- 172.21.15.112:0/2202156200 >> v1:172.21.15.112:6790/0 conn(0x7f260802a640 legacy=0x7f26080320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:07.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.432+0000 7f2615d15640 1 -- 172.21.15.112:0/2202156200 shutdown_connections 2024-03-29T10:56:07.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.432+0000 7f2615d15640 1 -- 172.21.15.112:0/2202156200 >> 172.21.15.112:0/2202156200 conn(0x7f26080058f0 msgr2=0x7f2608005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:07.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.433+0000 7f2615d15640 1 -- 172.21.15.112:0/2202156200 shutdown_connections 2024-03-29T10:56:07.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.433+0000 7f2615d15640 1 -- 172.21.15.112:0/2202156200 wait complete. 2024-03-29T10:56:07.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.435+0000 7f2615d15640 10 monclient: build_initial_monmap 2024-03-29T10:56:07.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.436+0000 7f2615d15640 10 monclient: monmap: 2024-03-29T10:56:07.416 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:56:07.416 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:07.416 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:56:07.416 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:56:07.416 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:07.416 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:07.417 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:56:07.417 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:56:07.417 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:56:07.417 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:07.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.436+0000 7f2615d15640 1 librados: starting msgr at 2024-03-29T10:56:07.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.436+0000 7f2615d15640 1 librados: starting objecter 2024-03-29T10:56:07.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.437+0000 7f2615d15640 1 Processor -- start 2024-03-29T10:56:07.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.437+0000 7f2615d15640 1 -- start start 2024-03-29T10:56:07.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.437+0000 7f2615d15640 1 librados: setting wanted keys 2024-03-29T10:56:07.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.437+0000 7f2615d15640 1 librados: calling monclient init 2024-03-29T10:56:07.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.437+0000 7f2615d15640 10 monclient: init 2024-03-29T10:56:07.419 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.438+0000 7f2615d15640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:07.419 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.438+0000 7f2615d15640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:56:07.420 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.438+0000 7f2615d15640 10 monclient(hunting): picked mon.noname-c con 0x7f260802a640 addr v1:172.21.15.112:6790/0 2024-03-29T10:56:07.420 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.439+0000 7f2614d13640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f260802a640 0x7f26080c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37376/0 (socket says 172.21.15.112:37376) 2024-03-29T10:56:07.420 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.439+0000 7f2615d15640 10 monclient(hunting): picked mon.noname-a con 0x7f26080327d0 addr v1:172.21.15.112:6789/0 2024-03-29T10:56:07.420 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.439+0000 7f2614d13640 1 -- 172.21.15.112:0/3678921855 learned_addr learned my addr 172.21.15.112:0/3678921855 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:56:07.423 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.439+0000 7f260d7fa640 10 client.?.objecter ms_handle_connect 0x7f260802a640 2024-03-29T10:56:07.423 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.442+0000 7f2615d15640 10 monclient(hunting): picked mon.noname-b con 0x7f2608034850 addr v1:172.21.15.118:6789/0 2024-03-29T10:56:07.423 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.442+0000 7f260d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:07.423 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.442+0000 7f2615d15640 1 -- 172.21.15.112:0/3678921855 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f260802cc40 con 0x7f26080327d0 2024-03-29T10:56:07.423 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.442+0000 7f2615d15640 1 -- 172.21.15.112:0/3678921855 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f260802cf20 con 0x7f2608034850 2024-03-29T10:56:07.423 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.442+0000 7f2615d15640 1 -- 172.21.15.112:0/3678921855 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26080bc150 con 0x7f260802a640 2024-03-29T10:56:07.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.442+0000 7f2615d15640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:07.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.442+0000 7f2615d15640 10 monclient(hunting): authenticate will time out at 1122.747144s 2024-03-29T10:56:07.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.444+0000 7f260d7fa640 10 client.?.objecter ms_handle_connect 0x7f26080327d0 2024-03-29T10:56:07.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.444+0000 7f260d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:07.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.444+0000 7f260d7fa640 1 -- 172.21.15.112:0/3678921855 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142432499 0 0) 0x7f260802cc40 con 0x7f26080327d0 2024-03-29T10:56:07.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.444+0000 7f260d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:07.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.444+0000 7f260d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:07.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.444+0000 7f260d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:07.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.444+0000 7f260d7fa640 10 monclient(hunting): my global_id is 4389 2024-03-29T10:56:07.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.444+0000 7f260d7fa640 1 -- 172.21.15.112:0/3678921855 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25f4004eb0 con 0x7f26080327d0 2024-03-29T10:56:07.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.444+0000 7f260d7fa640 10 client.?.objecter ms_handle_connect 0x7f2608034850 2024-03-29T10:56:07.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.444+0000 7f260d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:07.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.444+0000 7f260d7fa640 1 -- 172.21.15.112:0/3678921855 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 335060020 0 0) 0x7f25f4004eb0 con 0x7f26080327d0 2024-03-29T10:56:07.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f260d7fa640 1 -- 172.21.15.112:0/3678921855 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f260802cc40 con 0x7f26080327d0 2024-03-29T10:56:07.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f260d7fa640 1 -- 172.21.15.112:0/3678921855 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f2604001b70 con 0x7f26080327d0 2024-03-29T10:56:07.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f260d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:07.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f260d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:56:07.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f260d7fa640 10 monclient(hunting): dump: 2024-03-29T10:56:07.427 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:07.427 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:07.427 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:07.427 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:07.428 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:07.428 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:07.428 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:07.428 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:07.428 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:07.428 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:07.428 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f260d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:07.428 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f260d7fa640 1 -- 172.21.15.112:0/3678921855 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956848968 0 0) 0x7f26080bc150 con 0x7f260802a640 2024-03-29T10:56:07.428 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f260d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:07.428 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f260d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:07.428 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f260d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:07.428 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f260d7fa640 10 monclient(hunting): my global_id is 4409 2024-03-29T10:56:07.428 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f260d7fa640 1 -- 172.21.15.112:0/3678921855 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25f4004eb0 con 0x7f260802a640 2024-03-29T10:56:07.428 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f260d7fa640 1 -- 172.21.15.112:0/3678921855 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2217793284 0 0) 0x7f260802cc40 con 0x7f26080327d0 2024-03-29T10:56:07.428 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f260d7fa640 1 -- 172.21.15.112:0/3678921855 >> v1:172.21.15.112:6790/0 conn(0x7f260802a640 legacy=0x7f26080c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:07.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f260d7fa640 1 -- 172.21.15.112:0/3678921855 >> v1:172.21.15.118:6789/0 conn(0x7f2608034850 legacy=0x7f260815c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:07.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f260d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:56:07.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f260d7fa640 1 monclient: found mon.a 2024-03-29T10:56:07.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f260d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:07.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f260d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:56:07.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f260d7fa640 1 -- 172.21.15.112:0/3678921855 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f260815ceb0 con 0x7f26080327d0 2024-03-29T10:56:07.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f260d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:56:07.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f260d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:56:07.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f260d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:07.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f2615d15640 5 monclient: authenticate success, global_id 4389 2024-03-29T10:56:07.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f2615d15640 10 monclient: _renew_subs 2024-03-29T10:56:07.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f2615d15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:56:07.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f2615d15640 1 -- 172.21.15.112:0/3678921855 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26080bc680 con 0x7f26080327d0 2024-03-29T10:56:07.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f2615d15640 10 client.4389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:07.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f2615d15640 10 monclient: _renew_subs 2024-03-29T10:56:07.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f2615d15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:56:07.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.445+0000 7f2615d15640 1 -- 172.21.15.112:0/3678921855 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26080d2980 con 0x7f26080327d0 2024-03-29T10:56:07.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.446+0000 7f2615d15640 1 librados: init done 2024-03-29T10:56:07.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.446+0000 7f260d7fa640 1 -- 172.21.15.112:0/3678921855 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2604001cd0 con 0x7f26080327d0 2024-03-29T10:56:07.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.446+0000 7f260d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:07.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.446+0000 7f260effd640 4 set_mon_vals no callback set 2024-03-29T10:56:07.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.447+0000 7f260d7fa640 1 -- 172.21.15.112:0/3678921855 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f2604004d90 con 0x7f26080327d0 2024-03-29T10:56:07.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.447+0000 7f260d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:07.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.447+0000 7f260d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:07.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.447+0000 7f260d7fa640 10 monclient: dump: 2024-03-29T10:56:07.431 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:07.431 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:07.431 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:07.431 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:07.431 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:07.431 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:07.431 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:07.431 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:07.431 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:07.431 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:07.432 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.449+0000 7f260d7fa640 1 -- 172.21.15.112:0/3678921855 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f260401ad60 con 0x7f26080327d0 2024-03-29T10:56:07.432 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.450+0000 7f260d7fa640 1 -- 172.21.15.112:0/3678921855 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f2604084c90 con 0x7f26080327d0 2024-03-29T10:56:07.432 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.450+0000 7f260d7fa640 10 client.4389.objecter ms_dispatch 0x7f2610000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:07.432 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.450+0000 7f260d7fa640 3 client.4389.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:07.432 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.450+0000 7f260d7fa640 3 client.4389.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:07.432 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.450+0000 7f260d7fa640 20 client.4389.objecter dump_active .. 0 homeless 2024-03-29T10:56:07.432 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.451+0000 7f2614d13640 10 monclient: get_auth_request con 0x7f25f406fd20 auth_method 0 2024-03-29T10:56:07.432 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.451+0000 7f2615d15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:07.432 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.451+0000 7f2615d15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:07.432 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.451+0000 7f2615d15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:56:07.432 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.451+0000 7f2615d15640 1 -- 172.21.15.112:0/3678921855 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f0005080 con 0x7f26080327d0 2024-03-29T10:56:07.433 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.452+0000 7f260d7fa640 10 client.4389.objecter ms_handle_connect 0x7f25f406fd20 2024-03-29T10:56:07.441 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.460+0000 7f260d7fa640 1 -- 172.21.15.112:0/3678921855 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f26040503a0 con 0x7f26080327d0 2024-03-29T10:56:07.441 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.460+0000 7f260d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:07.441 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.460+0000 7f260d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:07.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.785+0000 7f2615d15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:07.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.785+0000 7f2615d15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:07.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.785+0000 7f2615d15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:56:07.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.785+0000 7f2615d15640 1 -- 172.21.15.112:0/3678921855 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25f0005330 con 0x7f26080327d0 2024-03-29T10:56:07.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.788+0000 7f260d7fa640 1 -- 172.21.15.112:0/3678921855 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f26040524a0 con 0x7f26080327d0 2024-03-29T10:56:07.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.788+0000 7f260d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:07.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.788+0000 7f260d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:07.770 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:56:07.770 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:07.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.791+0000 7f2615d15640 10 librados: watch_flush enter 2024-03-29T10:56:07.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.791+0000 7f2615d15640 10 librados: watch_flush exit 2024-03-29T10:56:07.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.791+0000 7f2615d15640 20 client.4389.objecter shutdown clearing up homeless session... 2024-03-29T10:56:07.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.791+0000 7f2615d15640 10 client.4389.objecter successfully canceled tick 2024-03-29T10:56:07.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.792+0000 7f2615d15640 1 -- 172.21.15.112:0/3678921855 >> v1:172.21.15.118:6816/3181073408 conn(0x7f25f406fd20 legacy=0x7f25f40721e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:07.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.792+0000 7f2615d15640 10 monclient: shutdown 2024-03-29T10:56:07.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.792+0000 7f2615d15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:07.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.792+0000 7f2615d15640 1 -- 172.21.15.112:0/3678921855 >> v1:172.21.15.112:6789/0 conn(0x7f26080327d0 legacy=0x7f26080bba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:07.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.792+0000 7f2615d15640 1 -- 172.21.15.112:0/3678921855 shutdown_connections 2024-03-29T10:56:07.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.792+0000 7f2615d15640 1 -- 172.21.15.112:0/3678921855 >> 172.21.15.112:0/3678921855 conn(0x7f26080058f0 msgr2=0x7f26080043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:07.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.792+0000 7f2615d15640 1 -- 172.21.15.112:0/3678921855 shutdown_connections 2024-03-29T10:56:07.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.793+0000 7f2615d15640 1 -- 172.21.15.112:0/3678921855 wait complete. 2024-03-29T10:56:07.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:07.793+0000 7f2615d15640 1 librados: shutdown 2024-03-29T10:56:07.790 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:14.597 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:14.748 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.767+0000 7fe1858ba640 10 monclient: get_monmap_and_config 2024-03-29T10:56:14.748 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.767+0000 7fe1858ba640 10 monclient: build_initial_monmap 2024-03-29T10:56:14.748 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.767+0000 7fe1858ba640 10 monclient: monmap: 2024-03-29T10:56:14.748 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:56:14.748 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:14.748 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:56:14.768281+0000 2024-03-29T10:56:14.748 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:56:14.768281+0000 2024-03-29T10:56:14.748 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:14.749 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:14.749 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:56:14.749 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:56:14.749 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:56:14.749 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:14.751 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.771+0000 7fe1858ba640 1 Processor -- start 2024-03-29T10:56:14.751 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.771+0000 7fe1858ba640 1 -- start start 2024-03-29T10:56:14.752 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.771+0000 7fe1858ba640 10 monclient: init 2024-03-29T10:56:14.752 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.772+0000 7fe1858ba640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:14.752 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.772+0000 7fe1858ba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:56:14.752 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.772+0000 7fe1858ba640 10 monclient(hunting): picked mon.noname-a con 0x7fe178006eb0 addr v1:172.21.15.112:6789/0 2024-03-29T10:56:14.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.772+0000 7fe1858ba640 10 monclient(hunting): picked mon.noname-c con 0x7fe178038f10 addr v1:172.21.15.118:6789/0 2024-03-29T10:56:14.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.772+0000 7fe1858ba640 10 monclient(hunting): picked mon.noname-b con 0x7fe178036690 addr v1:172.21.15.112:6790/0 2024-03-29T10:56:14.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.772+0000 7fe1858ba640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe178033c90 con 0x7fe178006eb0 2024-03-29T10:56:14.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.772+0000 7fe1858ba640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1780389f0 con 0x7fe178038f10 2024-03-29T10:56:14.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.772+0000 7fe1858ba640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe178030890 con 0x7fe178036690 2024-03-29T10:56:14.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.772+0000 7fe1858ba640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:14.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.772+0000 7fe1858ba640 10 monclient(hunting): authenticate will time out at 1130.076452s 2024-03-29T10:56:14.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.772+0000 7fe1850b9640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe178036690 0x7fe178036b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:60120/0 (socket says 172.21.15.112:60120) 2024-03-29T10:56:14.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.772+0000 7fe1850b9640 1 -- 172.21.15.112:0/3914994295 learned_addr learned my addr 172.21.15.112:0/3914994295 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:56:14.757 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.775+0000 7fe17f7fe640 1 -- 172.21.15.112:0/3914994295 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269934141 0 0) 0x7fe178030890 con 0x7fe178036690 2024-03-29T10:56:14.757 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.775+0000 7fe17f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:56:14.757 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.775+0000 7fe17f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:14.757 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.775+0000 7fe17f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:14.757 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.775+0000 7fe17f7fe640 10 monclient(hunting): my global_id is 4412 2024-03-29T10:56:14.757 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.775+0000 7fe17f7fe640 1 -- 172.21.15.112:0/3914994295 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe170003520 con 0x7fe178036690 2024-03-29T10:56:14.757 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.776+0000 7fe17f7fe640 1 -- 172.21.15.112:0/3914994295 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203231296 0 0) 0x7fe178033c90 con 0x7fe178006eb0 2024-03-29T10:56:14.758 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.776+0000 7fe17f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:56:14.758 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.776+0000 7fe17f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:14.758 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.776+0000 7fe17f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:14.758 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.776+0000 7fe17f7fe640 10 monclient(hunting): my global_id is 4392 2024-03-29T10:56:14.758 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.776+0000 7fe17f7fe640 1 -- 172.21.15.112:0/3914994295 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe178030890 con 0x7fe178006eb0 2024-03-29T10:56:14.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe17f7fe640 1 -- 172.21.15.112:0/3914994295 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1124918834 0 0) 0x7fe178030890 con 0x7fe178006eb0 2024-03-29T10:56:14.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe17f7fe640 1 -- 172.21.15.112:0/3914994295 >> v1:172.21.15.112:6790/0 conn(0x7fe178036690 legacy=0x7fe178036b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:14.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe17f7fe640 1 -- 172.21.15.112:0/3914994295 >> v1:172.21.15.118:6789/0 conn(0x7fe178038f10 legacy=0x7fe178039330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:56:14.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe17f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:56:14.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe17f7fe640 1 monclient: found mon.noname-a 2024-03-29T10:56:14.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe17f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:14.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe17f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T10:56:14.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe17f7fe640 1 -- 172.21.15.112:0/3914994295 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1780cbc40 con 0x7fe178006eb0 2024-03-29T10:56:14.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe17f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:56:14.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe17f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:56:14.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe17f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:14.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe17f7fe640 1 -- 172.21.15.112:0/3914994295 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe16c001690 con 0x7fe178006eb0 2024-03-29T10:56:14.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe17f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:14.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe17f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:56:14.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe17f7fe640 10 monclient: dump: 2024-03-29T10:56:14.761 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:14.761 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:14.761 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:14.761 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:14.761 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:14.762 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:14.762 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:14.762 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:14.762 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:14.762 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:14.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe1858ba640 5 monclient: authenticate success, global_id 4392 2024-03-29T10:56:14.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe1858ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:14.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe17f7fe640 1 -- 172.21.15.112:0/3914994295 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe16c001920 con 0x7fe178006eb0 2024-03-29T10:56:14.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe17f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:14.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe17f7fe640 1 -- 172.21.15.112:0/3914994295 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe16c001cb0 con 0x7fe178006eb0 2024-03-29T10:56:14.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe17f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:14.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe17f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:14.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe17f7fe640 10 monclient: dump: 2024-03-29T10:56:14.762 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:14.763 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:14.763 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:14.763 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:14.763 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:14.763 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:14.763 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:14.763 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:14.763 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:14.763 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:14.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe1858ba640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:14.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.777+0000 7fe1858ba640 4 set_mon_vals no callback set 2024-03-29T10:56:14.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.778+0000 7fe1858ba640 10 monclient: shutdown 2024-03-29T10:56:14.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.778+0000 7fe1858ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:14.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.778+0000 7fe1858ba640 1 -- 172.21.15.112:0/3914994295 >> v1:172.21.15.112:6789/0 conn(0x7fe178006eb0 legacy=0x7fe178031760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:14.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.779+0000 7fe1858ba640 1 -- 172.21.15.112:0/3914994295 shutdown_connections 2024-03-29T10:56:14.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.779+0000 7fe1858ba640 1 -- 172.21.15.112:0/3914994295 >> 172.21.15.112:0/3914994295 conn(0x7fe178005c20 msgr2=0x7fe178006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:14.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.779+0000 7fe1858ba640 1 -- 172.21.15.112:0/3914994295 shutdown_connections 2024-03-29T10:56:14.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.779+0000 7fe1858ba640 1 -- 172.21.15.112:0/3914994295 wait complete. 2024-03-29T10:56:14.771 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.780+0000 7fe1858ba640 10 monclient: build_initial_monmap 2024-03-29T10:56:14.771 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.780+0000 7fe1858ba640 10 monclient: monmap: 2024-03-29T10:56:14.771 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:56:14.771 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:14.771 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:56:14.771 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:56:14.772 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:14.772 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:14.772 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:56:14.772 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:56:14.772 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:56:14.772 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:14.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.781+0000 7fe1858ba640 1 librados: starting msgr at 2024-03-29T10:56:14.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.781+0000 7fe1858ba640 1 librados: starting objecter 2024-03-29T10:56:14.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.781+0000 7fe1858ba640 1 Processor -- start 2024-03-29T10:56:14.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.781+0000 7fe1858ba640 1 -- start start 2024-03-29T10:56:14.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.781+0000 7fe1858ba640 1 librados: setting wanted keys 2024-03-29T10:56:14.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.781+0000 7fe1858ba640 1 librados: calling monclient init 2024-03-29T10:56:14.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.781+0000 7fe1858ba640 10 monclient: init 2024-03-29T10:56:14.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.782+0000 7fe1858ba640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:14.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.782+0000 7fe1858ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:56:14.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.782+0000 7fe1858ba640 10 monclient(hunting): picked mon.noname-b con 0x7fe178006eb0 addr v1:172.21.15.118:6789/0 2024-03-29T10:56:14.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.782+0000 7fe1858ba640 10 monclient(hunting): picked mon.noname-a con 0x7fe178036690 addr v1:172.21.15.112:6789/0 2024-03-29T10:56:14.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.782+0000 7fe1858ba640 10 monclient(hunting): picked mon.noname-c con 0x7fe178038f10 addr v1:172.21.15.112:6790/0 2024-03-29T10:56:14.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.782+0000 7fe1858ba640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe17800b400 con 0x7fe178036690 2024-03-29T10:56:14.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.782+0000 7fe1858ba640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe17800b6e0 con 0x7fe178006eb0 2024-03-29T10:56:14.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.782+0000 7fe1858ba640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe17800a0c0 con 0x7fe178038f10 2024-03-29T10:56:14.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.782+0000 7fe1858ba640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:14.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.782+0000 7fe1858ba640 10 monclient(hunting): authenticate will time out at 1130.086507s 2024-03-29T10:56:14.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.782+0000 7fe1850b9640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe178038f10 0x7fe178174cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:60780/0 (socket says 172.21.15.112:60780) 2024-03-29T10:56:14.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.782+0000 7fe1850b9640 1 -- 172.21.15.112:0/855505941 learned_addr learned my addr 172.21.15.112:0/855505941 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:56:14.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.783+0000 7fe17d7fa640 10 client.?.objecter ms_handle_connect 0x7fe178038f10 2024-03-29T10:56:14.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.783+0000 7fe17d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:14.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.783+0000 7fe17d7fa640 1 -- 172.21.15.112:0/855505941 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1446899816 0 0) 0x7fe17800a0c0 con 0x7fe178038f10 2024-03-29T10:56:14.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.783+0000 7fe17d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:14.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.783+0000 7fe17d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:14.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.783+0000 7fe17d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:14.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.783+0000 7fe17d7fa640 10 monclient(hunting): my global_id is 4415 2024-03-29T10:56:14.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.783+0000 7fe17d7fa640 1 -- 172.21.15.112:0/855505941 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe164004eb0 con 0x7fe178038f10 2024-03-29T10:56:14.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.784+0000 7fe17d7fa640 1 -- 172.21.15.112:0/855505941 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1608229812 0 0) 0x7fe164004eb0 con 0x7fe178038f10 2024-03-29T10:56:14.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.784+0000 7fe17d7fa640 1 -- 172.21.15.112:0/855505941 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe17800a0c0 con 0x7fe178038f10 2024-03-29T10:56:14.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.785+0000 7fe17d7fa640 1 -- 172.21.15.112:0/855505941 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe18004fdd0 con 0x7fe178038f10 2024-03-29T10:56:14.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.785+0000 7fe17d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:14.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.785+0000 7fe17d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:14.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.785+0000 7fe17d7fa640 10 monclient(hunting): dump: 2024-03-29T10:56:14.775 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:14.775 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:14.775 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:14.775 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:14.775 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:14.775 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:14.775 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:14.775 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:14.775 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:14.775 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:14.775 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.785+0000 7fe17d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:14.775 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.785+0000 7fe17d7fa640 1 -- 172.21.15.112:0/855505941 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 163915505 0 0) 0x7fe17800a0c0 con 0x7fe178038f10 2024-03-29T10:56:14.775 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.785+0000 7fe17d7fa640 1 -- 172.21.15.112:0/855505941 >> v1:172.21.15.118:6789/0 conn(0x7fe178006eb0 legacy=0x7fe178169390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:14.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.785+0000 7fe17d7fa640 1 -- 172.21.15.112:0/855505941 >> v1:172.21.15.112:6789/0 conn(0x7fe178036690 legacy=0x7fe17800ceb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:14.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.785+0000 7fe17d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:56:14.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.785+0000 7fe17d7fa640 1 monclient: found mon.c 2024-03-29T10:56:14.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.786+0000 7fe17d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:14.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.786+0000 7fe17d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:14.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.786+0000 7fe17d7fa640 1 -- 172.21.15.112:0/855505941 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17800a5e0 con 0x7fe178038f10 2024-03-29T10:56:14.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.786+0000 7fe17d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:56:14.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.786+0000 7fe17d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:56:14.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.786+0000 7fe17d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:14.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.786+0000 7fe1858ba640 5 monclient: authenticate success, global_id 4415 2024-03-29T10:56:14.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.786+0000 7fe1858ba640 10 monclient: _renew_subs 2024-03-29T10:56:14.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.786+0000 7fe1858ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:14.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.786+0000 7fe1858ba640 1 -- 172.21.15.112:0/855505941 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe17802f870 con 0x7fe178038f10 2024-03-29T10:56:14.777 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.786+0000 7fe1858ba640 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:14.777 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.786+0000 7fe1858ba640 10 monclient: _renew_subs 2024-03-29T10:56:14.777 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.786+0000 7fe1858ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:14.777 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.786+0000 7fe1858ba640 1 -- 172.21.15.112:0/855505941 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe178169aa0 con 0x7fe178038f10 2024-03-29T10:56:14.777 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.786+0000 7fe1858ba640 1 librados: init done 2024-03-29T10:56:14.777 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.787+0000 7fe17d7fa640 1 -- 172.21.15.112:0/855505941 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe18004f400 con 0x7fe178038f10 2024-03-29T10:56:14.777 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.788+0000 7fe17d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:14.777 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.788+0000 7fe17d7fa640 1 -- 172.21.15.112:0/855505941 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe180075f10 con 0x7fe178038f10 2024-03-29T10:56:14.777 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.788+0000 7fe17d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:14.777 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.788+0000 7fe17d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:14.777 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.788+0000 7fe17d7fa640 10 monclient: dump: 2024-03-29T10:56:14.777 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:14.777 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:14.777 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:14.778 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:14.778 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:14.778 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:14.778 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:14.778 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:14.778 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:14.778 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:14.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.788+0000 7fe1858ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:14.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.788+0000 7fe1858ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:14.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.788+0000 7fe1858ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:14.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.788+0000 7fe1858ba640 1 -- 172.21.15.112:0/855505941 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe178007b90 con 0x7fe178038f10 2024-03-29T10:56:14.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.789+0000 7fe17effd640 4 set_mon_vals no callback set 2024-03-29T10:56:14.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.793+0000 7fe17d7fa640 1 -- 172.21.15.112:0/855505941 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fe180067bf0 con 0x7fe178038f10 2024-03-29T10:56:14.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.794+0000 7fe17d7fa640 1 -- 172.21.15.112:0/855505941 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7fe1800ff580 con 0x7fe178038f10 2024-03-29T10:56:14.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.794+0000 7fe17d7fa640 10 client.4415.objecter ms_dispatch 0x7fe180000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:14.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.794+0000 7fe17d7fa640 3 client.4415.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:14.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.794+0000 7fe17d7fa640 3 client.4415.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:14.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.794+0000 7fe17d7fa640 20 client.4415.objecter dump_active .. 0 homeless 2024-03-29T10:56:14.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.796+0000 7fe1848b8640 10 monclient: get_auth_request con 0x7fe16406dc20 auth_method 0 2024-03-29T10:56:14.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.796+0000 7fe17d7fa640 10 client.4415.objecter ms_handle_connect 0x7fe16406dc20 2024-03-29T10:56:14.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.802+0000 7fe17d7fa640 1 -- 172.21.15.112:0/855505941 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe1800cae60 con 0x7fe178038f10 2024-03-29T10:56:14.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.802+0000 7fe17d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:14.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:14.802+0000 7fe17d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:15.115 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:15.132+0000 7fe1858ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:15.115 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:15.132+0000 7fe1858ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:15.115 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:15.132+0000 7fe1858ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:15.115 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:15.132+0000 7fe1858ba640 1 -- 172.21.15.112:0/855505941 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe178004cd0 con 0x7fe178038f10 2024-03-29T10:56:15.116 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:15.135+0000 7fe17d7fa640 1 -- 172.21.15.112:0/855505941 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7fe1800ccd90 con 0x7fe178038f10 2024-03-29T10:56:15.116 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:15.135+0000 7fe17d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:15.116 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:15.135+0000 7fe17d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:15.117 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:56:15.117 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:15.120 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:15.139+0000 7fe1858ba640 10 librados: watch_flush enter 2024-03-29T10:56:15.120 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:15.139+0000 7fe1858ba640 10 librados: watch_flush exit 2024-03-29T10:56:15.120 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:15.139+0000 7fe1858ba640 20 client.4415.objecter shutdown clearing up homeless session... 2024-03-29T10:56:15.120 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:15.139+0000 7fe1858ba640 10 client.4415.objecter successfully canceled tick 2024-03-29T10:56:15.120 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:15.140+0000 7fe1858ba640 1 -- 172.21.15.112:0/855505941 >> v1:172.21.15.118:6816/3181073408 conn(0x7fe16406dc20 legacy=0x7fe1640700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:15.120 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:15.140+0000 7fe1858ba640 10 monclient: shutdown 2024-03-29T10:56:15.120 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:15.140+0000 7fe1858ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:15.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:15.140+0000 7fe1858ba640 1 -- 172.21.15.112:0/855505941 >> v1:172.21.15.112:6790/0 conn(0x7fe178038f10 legacy=0x7fe178174cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:15.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:15.140+0000 7fe1858ba640 1 -- 172.21.15.112:0/855505941 shutdown_connections 2024-03-29T10:56:15.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:15.140+0000 7fe1858ba640 1 -- 172.21.15.112:0/855505941 >> 172.21.15.112:0/855505941 conn(0x7fe178005c20 msgr2=0x7fe1780043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:15.122 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:15.141+0000 7fe1858ba640 1 -- 172.21.15.112:0/855505941 shutdown_connections 2024-03-29T10:56:15.122 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:15.141+0000 7fe1858ba640 1 -- 172.21.15.112:0/855505941 wait complete. 2024-03-29T10:56:15.122 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:15.141+0000 7fe1858ba640 1 librados: shutdown 2024-03-29T10:56:15.140 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:21.946 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:22.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.118+0000 7f1975120640 10 monclient: get_monmap_and_config 2024-03-29T10:56:22.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.118+0000 7f1975120640 10 monclient: build_initial_monmap 2024-03-29T10:56:22.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.118+0000 7f1975120640 10 monclient: monmap: 2024-03-29T10:56:22.099 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:56:22.099 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:22.099 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:56:22.119109+0000 2024-03-29T10:56:22.099 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:56:22.119109+0000 2024-03-29T10:56:22.099 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:22.099 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:22.099 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:56:22.099 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:56:22.099 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:56:22.099 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:22.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.119+0000 7f1975120640 1 Processor -- start 2024-03-29T10:56:22.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.119+0000 7f1975120640 1 -- start start 2024-03-29T10:56:22.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.119+0000 7f1975120640 10 monclient: init 2024-03-29T10:56:22.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.120+0000 7f1975120640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:22.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.120+0000 7f1975120640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:56:22.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.120+0000 7f196ffff640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f19700b2800 0x7f1970142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35010/0 (socket says 172.21.15.112:35010) 2024-03-29T10:56:22.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.120+0000 7f196ffff640 1 -- 172.21.15.112:0/4046546802 learned_addr learned my addr 172.21.15.112:0/4046546802 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:56:22.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.120+0000 7f1975120640 10 monclient(hunting): picked mon.noname-b con 0x7f19700b2800 addr v1:172.21.15.112:6790/0 2024-03-29T10:56:22.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.120+0000 7f1975120640 10 monclient(hunting): picked mon.noname-a con 0x7f19700a9c00 addr v1:172.21.15.112:6789/0 2024-03-29T10:56:22.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.120+0000 7f1975120640 10 monclient(hunting): picked mon.noname-c con 0x7f19700b38c0 addr v1:172.21.15.118:6789/0 2024-03-29T10:56:22.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.120+0000 7f1975120640 1 -- 172.21.15.112:0/4046546802 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1970141eb0 con 0x7f19700a9c00 2024-03-29T10:56:22.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.120+0000 7f1975120640 1 -- 172.21.15.112:0/4046546802 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19700b22d0 con 0x7f19700b38c0 2024-03-29T10:56:22.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.120+0000 7f1975120640 1 -- 172.21.15.112:0/4046546802 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1970142d30 con 0x7f19700b2800 2024-03-29T10:56:22.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.120+0000 7f1975120640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:22.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.120+0000 7f1975120640 10 monclient(hunting): authenticate will time out at 1137.424710s 2024-03-29T10:56:22.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.123+0000 7f196effd640 1 -- 172.21.15.112:0/4046546802 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457347377 0 0) 0x7f1970142d30 con 0x7f19700b2800 2024-03-29T10:56:22.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.123+0000 7f196effd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:22.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.123+0000 7f196effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:22.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.123+0000 7f196effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:22.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.123+0000 7f196effd640 10 monclient(hunting): my global_id is 4418 2024-03-29T10:56:22.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.123+0000 7f196effd640 1 -- 172.21.15.112:0/4046546802 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1958003520 con 0x7f19700b2800 2024-03-29T10:56:22.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.123+0000 7f196effd640 1 -- 172.21.15.112:0/4046546802 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3272206903 0 0) 0x7f1970141eb0 con 0x7f19700a9c00 2024-03-29T10:56:22.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.123+0000 7f196effd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:22.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.123+0000 7f196effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:22.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.123+0000 7f196effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:22.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.123+0000 7f196effd640 10 monclient(hunting): my global_id is 4395 2024-03-29T10:56:22.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.123+0000 7f196effd640 1 -- 172.21.15.112:0/4046546802 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1970142d30 con 0x7f19700a9c00 2024-03-29T10:56:22.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.124+0000 7f196effd640 1 -- 172.21.15.112:0/4046546802 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 46511875 0 0) 0x7f1958003520 con 0x7f19700b2800 2024-03-29T10:56:22.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.124+0000 7f196effd640 1 -- 172.21.15.112:0/4046546802 >> v1:172.21.15.118:6789/0 conn(0x7f19700b38c0 legacy=0x7f197014d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:22.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.124+0000 7f196effd640 1 -- 172.21.15.112:0/4046546802 >> v1:172.21.15.112:6789/0 conn(0x7f19700a9c00 legacy=0x7f197013bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:22.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.124+0000 7f196effd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:22.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.124+0000 7f196effd640 1 monclient: found mon.noname-b 2024-03-29T10:56:22.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.124+0000 7f196effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:22.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.124+0000 7f196effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:56:22.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.124+0000 7f196effd640 1 -- 172.21.15.112:0/4046546802 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19700aaf90 con 0x7f19700b2800 2024-03-29T10:56:22.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.124+0000 7f196effd640 10 monclient: _finish_auth 0 2024-03-29T10:56:22.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.124+0000 7f196effd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:22.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.124+0000 7f196effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:22.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.124+0000 7f196effd640 1 -- 172.21.15.112:0/4046546802 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1960001690 con 0x7f19700b2800 2024-03-29T10:56:22.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.124+0000 7f196effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:22.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.124+0000 7f196effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:22.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.124+0000 7f196effd640 10 monclient: dump: 2024-03-29T10:56:22.108 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:22.109 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:22.109 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:22.109 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:22.109 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:22.109 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:22.109 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:22.109 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:22.109 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:22.109 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:22.109 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.125+0000 7f1975120640 5 monclient: authenticate success, global_id 4418 2024-03-29T10:56:22.109 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.125+0000 7f1975120640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:22.109 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.126+0000 7f196effd640 1 -- 172.21.15.112:0/4046546802 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19600017f0 con 0x7f19700b2800 2024-03-29T10:56:22.109 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.126+0000 7f196effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:22.109 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.126+0000 7f196effd640 1 -- 172.21.15.112:0/4046546802 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1960001bc0 con 0x7f19700b2800 2024-03-29T10:56:22.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.126+0000 7f196effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:22.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.126+0000 7f196effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:22.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.126+0000 7f196effd640 10 monclient: dump: 2024-03-29T10:56:22.110 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:22.110 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:22.110 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:22.110 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:22.110 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:22.110 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:22.110 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:22.110 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:22.110 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:22.110 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:22.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.126+0000 7f1975120640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:22.111 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.126+0000 7f1975120640 4 set_mon_vals no callback set 2024-03-29T10:56:22.112 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.127+0000 7f1975120640 10 monclient: shutdown 2024-03-29T10:56:22.112 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.127+0000 7f1975120640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:22.112 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.127+0000 7f1975120640 1 -- 172.21.15.112:0/4046546802 >> v1:172.21.15.112:6790/0 conn(0x7f19700b2800 legacy=0x7f1970142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:22.112 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.127+0000 7f1975120640 1 -- 172.21.15.112:0/4046546802 shutdown_connections 2024-03-29T10:56:22.112 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.127+0000 7f1975120640 1 -- 172.21.15.112:0/4046546802 >> 172.21.15.112:0/4046546802 conn(0x7f1970066a00 msgr2=0x7f1970066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:22.112 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.128+0000 7f1975120640 1 -- 172.21.15.112:0/4046546802 shutdown_connections 2024-03-29T10:56:22.112 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.129+0000 7f1975120640 1 -- 172.21.15.112:0/4046546802 wait complete. 2024-03-29T10:56:22.112 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.130+0000 7f1975120640 10 monclient: build_initial_monmap 2024-03-29T10:56:22.112 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.130+0000 7f1975120640 10 monclient: monmap: 2024-03-29T10:56:22.112 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:56:22.112 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:22.113 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:56:22.113 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:56:22.113 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:22.113 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:22.113 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:56:22.113 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:56:22.113 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:56:22.113 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:22.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.130+0000 7f1975120640 1 librados: starting msgr at 2024-03-29T10:56:22.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.130+0000 7f1975120640 1 librados: starting objecter 2024-03-29T10:56:22.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.130+0000 7f1975120640 1 Processor -- start 2024-03-29T10:56:22.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.131+0000 7f1975120640 1 -- start start 2024-03-29T10:56:22.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.131+0000 7f1975120640 1 librados: setting wanted keys 2024-03-29T10:56:22.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.131+0000 7f1975120640 1 librados: calling monclient init 2024-03-29T10:56:22.114 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.131+0000 7f1975120640 10 monclient: init 2024-03-29T10:56:22.114 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.132+0000 7f1975120640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:22.114 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.132+0000 7f1975120640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:56:22.114 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.132+0000 7f1975120640 10 monclient(hunting): picked mon.noname-c con 0x7f19700a9c00 addr v1:172.21.15.112:6790/0 2024-03-29T10:56:22.116 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.133+0000 7f1975120640 10 monclient(hunting): picked mon.noname-a con 0x7f19700b2800 addr v1:172.21.15.112:6789/0 2024-03-29T10:56:22.116 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.133+0000 7f196ffff640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f19700a9c00 0x7f19700adbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:36524/0 (socket says 172.21.15.112:36524) 2024-03-29T10:56:22.117 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.136+0000 7f196ffff640 1 -- 172.21.15.112:0/75650399 learned_addr learned my addr 172.21.15.112:0/75650399 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:56:22.117 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.136+0000 7f196cff9640 10 client.?.objecter ms_handle_connect 0x7f19700a9c00 2024-03-29T10:56:22.117 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.137+0000 7f196cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:22.118 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.137+0000 7f196cff9640 10 client.?.objecter ms_handle_connect 0x7f19700b2800 2024-03-29T10:56:22.118 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.137+0000 7f196cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:22.118 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.137+0000 7f1975120640 10 monclient(hunting): picked mon.noname-b con 0x7f19700b38c0 addr v1:172.21.15.118:6789/0 2024-03-29T10:56:22.118 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.137+0000 7f1975120640 1 -- 172.21.15.112:0/75650399 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19700aef90 con 0x7f19700b2800 2024-03-29T10:56:22.118 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.137+0000 7f1975120640 1 -- 172.21.15.112:0/75650399 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19700af200 con 0x7f19700b38c0 2024-03-29T10:56:22.118 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.137+0000 7f1975120640 1 -- 172.21.15.112:0/75650399 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19700af4e0 con 0x7f19700a9c00 2024-03-29T10:56:22.118 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.137+0000 7f1975120640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:22.118 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.137+0000 7f1975120640 10 monclient(hunting): authenticate will time out at 1137.441515s 2024-03-29T10:56:22.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 1 -- 172.21.15.112:0/75650399 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4022361904 0 0) 0x7f19700aef90 con 0x7f19700b2800 2024-03-29T10:56:22.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:22.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:22.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:22.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 10 monclient(hunting): my global_id is 4398 2024-03-29T10:56:22.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 1 -- 172.21.15.112:0/75650399 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f195c004eb0 con 0x7f19700b2800 2024-03-29T10:56:22.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 1 -- 172.21.15.112:0/75650399 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528645618 0 0) 0x7f19700af4e0 con 0x7f19700a9c00 2024-03-29T10:56:22.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:22.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:22.122 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:22.122 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 10 monclient(hunting): my global_id is 4421 2024-03-29T10:56:22.122 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 1 -- 172.21.15.112:0/75650399 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19700aef90 con 0x7f19700a9c00 2024-03-29T10:56:22.122 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 1 -- 172.21.15.112:0/75650399 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1139238375 0 0) 0x7f19700aef90 con 0x7f19700a9c00 2024-03-29T10:56:22.122 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 1 -- 172.21.15.112:0/75650399 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19700af4e0 con 0x7f19700a9c00 2024-03-29T10:56:22.122 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 1 -- 172.21.15.112:0/75650399 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2929086956 0 0) 0x7f195c004eb0 con 0x7f19700b2800 2024-03-29T10:56:22.122 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 1 -- 172.21.15.112:0/75650399 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19700aef90 con 0x7f19700b2800 2024-03-29T10:56:22.123 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 1 -- 172.21.15.112:0/75650399 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1960001b00 con 0x7f19700a9c00 2024-03-29T10:56:22.123 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:22.123 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:22.123 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 10 monclient(hunting): dump: 2024-03-29T10:56:22.123 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:22.123 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:22.123 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:22.123 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:22.123 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:22.123 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:22.123 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:22.123 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:22.124 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:22.124 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:22.124 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:22.124 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 1 -- 172.21.15.112:0/75650399 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f19640017a0 con 0x7f19700b2800 2024-03-29T10:56:22.124 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:22.124 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:22.124 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 10 monclient(hunting): dump: 2024-03-29T10:56:22.124 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:22.124 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:22.124 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:22.124 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:22.124 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:22.124 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:22.124 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:22.124 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:22.125 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:22.125 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:22.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.140+0000 7f196cff9640 1 -- 172.21.15.112:0/75650399 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3325052663 0 0) 0x7f19700af4e0 con 0x7f19700a9c00 2024-03-29T10:56:22.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f196cff9640 1 -- 172.21.15.112:0/75650399 >> v1:172.21.15.118:6789/0 conn(0x7f19700b38c0 legacy=0x7f19701e2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:22.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f196cff9640 1 -- 172.21.15.112:0/75650399 >> v1:172.21.15.112:6789/0 conn(0x7f19700b2800 legacy=0x7f1970139f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:22.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f196cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:56:22.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f196cff9640 1 monclient: found mon.c 2024-03-29T10:56:22.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f196cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:22.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f196cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:22.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f196cff9640 1 -- 172.21.15.112:0/75650399 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19700ad430 con 0x7f19700a9c00 2024-03-29T10:56:22.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f196cff9640 10 monclient: _finish_auth 0 2024-03-29T10:56:22.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f196cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:56:22.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f196cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:22.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f196cff9640 10 client.?.objecter ms_handle_connect 0x7f19700b38c0 2024-03-29T10:56:22.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f196cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:22.126 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f196cff9640 1 -- 172.21.15.112:0/75650399 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1960004a90 con 0x7f19700a9c00 2024-03-29T10:56:22.126 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f196cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:22.126 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f196cff9640 1 -- 172.21.15.112:0/75650399 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1960004da0 con 0x7f19700a9c00 2024-03-29T10:56:22.126 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f196cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:22.126 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f196cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:22.126 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f196cff9640 10 monclient: dump: 2024-03-29T10:56:22.126 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:22.126 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:22.126 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:22.126 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:22.126 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:22.126 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:22.127 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:22.127 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:22.127 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:22.127 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:22.129 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f196e7fc640 4 set_mon_vals no callback set 2024-03-29T10:56:22.129 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f1975120640 5 monclient: authenticate success, global_id 4421 2024-03-29T10:56:22.129 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f1975120640 10 monclient: _renew_subs 2024-03-29T10:56:22.129 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f1975120640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:22.129 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f1975120640 1 -- 172.21.15.112:0/75650399 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19700ae2d0 con 0x7f19700a9c00 2024-03-29T10:56:22.130 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f1975120640 10 client.4421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:22.130 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f1975120640 10 monclient: _renew_subs 2024-03-29T10:56:22.130 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f1975120640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:22.130 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f1975120640 1 -- 172.21.15.112:0/75650399 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19700ab550 con 0x7f19700a9c00 2024-03-29T10:56:22.130 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.141+0000 7f1975120640 1 librados: init done 2024-03-29T10:56:22.130 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.144+0000 7f19567fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:22.130 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.145+0000 7f19567fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:22.130 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.145+0000 7f19567fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:22.130 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.145+0000 7f19567fc640 1 -- 172.21.15.112:0/75650399 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f197004f730 con 0x7f19700a9c00 2024-03-29T10:56:22.130 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.146+0000 7f196cff9640 1 -- 172.21.15.112:0/75650399 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f196001ad70 con 0x7f19700a9c00 2024-03-29T10:56:22.130 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.147+0000 7f196cff9640 1 -- 172.21.15.112:0/75650399 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f1960084b30 con 0x7f19700a9c00 2024-03-29T10:56:22.130 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.147+0000 7f196cff9640 10 client.4421.objecter ms_dispatch 0x7f1970000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:22.130 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.147+0000 7f196cff9640 3 client.4421.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:22.130 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.147+0000 7f196cff9640 3 client.4421.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:22.131 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.147+0000 7f196cff9640 20 client.4421.objecter dump_active .. 0 homeless 2024-03-29T10:56:22.131 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.147+0000 7f196f7fe640 10 monclient: get_auth_request con 0x7f195c06de20 auth_method 0 2024-03-29T10:56:22.131 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.147+0000 7f196cff9640 10 client.4421.objecter ms_handle_connect 0x7f195c06de20 2024-03-29T10:56:22.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.155+0000 7f196cff9640 1 -- 172.21.15.112:0/75650399 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1960050410 con 0x7f19700a9c00 2024-03-29T10:56:22.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.155+0000 7f196cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:22.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.155+0000 7f196cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:22.458 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.476+0000 7f19567fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:22.458 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.476+0000 7f19567fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:22.458 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.476+0000 7f19567fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:22.458 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.476+0000 7f19567fc640 1 -- 172.21.15.112:0/75650399 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19700ad5c0 con 0x7f19700a9c00 2024-03-29T10:56:22.460 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.480+0000 7f196cff9640 1 -- 172.21.15.112:0/75650399 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f1960052340 con 0x7f19700a9c00 2024-03-29T10:56:22.461 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.480+0000 7f196cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:22.461 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.480+0000 7f196cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:22.461 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:56:22.461 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:22.464 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.484+0000 7f19567fc640 10 librados: watch_flush enter 2024-03-29T10:56:22.464 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.484+0000 7f19567fc640 10 librados: watch_flush exit 2024-03-29T10:56:22.465 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.484+0000 7f19567fc640 20 client.4421.objecter shutdown clearing up homeless session... 2024-03-29T10:56:22.465 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.484+0000 7f19567fc640 10 client.4421.objecter successfully canceled tick 2024-03-29T10:56:22.466 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.485+0000 7f19567fc640 1 -- 172.21.15.112:0/75650399 >> v1:172.21.15.118:6816/3181073408 conn(0x7f195c06de20 legacy=0x7f195c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:22.466 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.485+0000 7f19567fc640 10 monclient: shutdown 2024-03-29T10:56:22.466 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.485+0000 7f19567fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:22.466 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.485+0000 7f19567fc640 1 -- 172.21.15.112:0/75650399 >> v1:172.21.15.112:6790/0 conn(0x7f19700a9c00 legacy=0x7f19700adbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:22.467 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.486+0000 7f19567fc640 1 -- 172.21.15.112:0/75650399 shutdown_connections 2024-03-29T10:56:22.467 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.486+0000 7f19567fc640 1 -- 172.21.15.112:0/75650399 >> 172.21.15.112:0/75650399 conn(0x7f1970066a00 msgr2=0x7f1970140590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:22.467 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.486+0000 7f19567fc640 1 -- 172.21.15.112:0/75650399 shutdown_connections 2024-03-29T10:56:22.467 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.487+0000 7f19567fc640 1 -- 172.21.15.112:0/75650399 wait complete. 2024-03-29T10:56:22.467 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:22.487+0000 7f19567fc640 1 librados: shutdown 2024-03-29T10:56:22.484 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:29.290 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:29.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.476+0000 7f59dfac1640 10 monclient: get_monmap_and_config 2024-03-29T10:56:29.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.476+0000 7f59dfac1640 10 monclient: build_initial_monmap 2024-03-29T10:56:29.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.476+0000 7f59dfac1640 10 monclient: monmap: 2024-03-29T10:56:29.457 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:56:29.457 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:29.457 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:56:29.477018+0000 2024-03-29T10:56:29.457 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:56:29.477018+0000 2024-03-29T10:56:29.457 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:29.457 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:29.457 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:56:29.458 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:56:29.458 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:56:29.458 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:29.460 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.478+0000 7f59dfac1640 1 Processor -- start 2024-03-29T10:56:29.460 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.479+0000 7f59dfac1640 1 -- start start 2024-03-29T10:56:29.460 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.479+0000 7f59dfac1640 10 monclient: init 2024-03-29T10:56:29.461 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.480+0000 7f59dfac1640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:29.461 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.480+0000 7f59dfac1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:29.461 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.480+0000 7f59dfac1640 10 monclient(hunting): picked mon.noname-b con 0x7f59d80b0860 addr v1:172.21.15.112:6790/0 2024-03-29T10:56:29.461 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.480+0000 7f59dfac1640 10 monclient(hunting): picked mon.noname-c con 0x7f59d813ef30 addr v1:172.21.15.118:6789/0 2024-03-29T10:56:29.461 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.480+0000 7f59dfac1640 10 monclient(hunting): picked mon.noname-a con 0x7f59d80af8e0 addr v1:172.21.15.112:6789/0 2024-03-29T10:56:29.461 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.480+0000 7f59dfac1640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d80b0c40 con 0x7f59d80af8e0 2024-03-29T10:56:29.461 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.480+0000 7f59dfac1640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d80aed10 con 0x7f59d813ef30 2024-03-29T10:56:29.461 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.480+0000 7f59dfac1640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d80afd50 con 0x7f59d80b0860 2024-03-29T10:56:29.462 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.480+0000 7f59dfac1640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:29.462 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.480+0000 7f59dfac1640 10 monclient(hunting): authenticate will time out at 1144.784302s 2024-03-29T10:56:29.462 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.480+0000 7f59dd85d640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f59d80b0860 0x7f59d813e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:52066/0 (socket says 172.21.15.112:52066) 2024-03-29T10:56:29.462 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.480+0000 7f59dd85d640 1 -- 172.21.15.112:0/3971234663 learned_addr learned my addr 172.21.15.112:0/3971234663 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:56:29.463 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.483+0000 7f59dc85b640 1 -- 172.21.15.112:0/3971234663 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811927250 0 0) 0x7f59d80afd50 con 0x7f59d80b0860 2024-03-29T10:56:29.464 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.483+0000 7f59dc85b640 10 monclient(hunting): _negotiate 2024-03-29T10:56:29.464 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.483+0000 7f59dc85b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:29.464 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.483+0000 7f59dc85b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:29.464 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.483+0000 7f59dc85b640 10 monclient(hunting): my global_id is 4424 2024-03-29T10:56:29.464 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.483+0000 7f59dc85b640 1 -- 172.21.15.112:0/3971234663 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59c4003520 con 0x7f59d80b0860 2024-03-29T10:56:29.465 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 1 -- 172.21.15.112:0/3971234663 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2475988906 0 0) 0x7f59d80b0c40 con 0x7f59d80af8e0 2024-03-29T10:56:29.465 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 10 monclient(hunting): _negotiate 2024-03-29T10:56:29.465 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:29.465 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:29.465 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 10 monclient(hunting): my global_id is 4401 2024-03-29T10:56:29.465 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 1 -- 172.21.15.112:0/3971234663 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59d80afd50 con 0x7f59d80af8e0 2024-03-29T10:56:29.465 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 1 -- 172.21.15.112:0/3971234663 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2121239349 0 0) 0x7f59c4003520 con 0x7f59d80b0860 2024-03-29T10:56:29.465 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 1 -- 172.21.15.112:0/3971234663 >> v1:172.21.15.118:6789/0 conn(0x7f59d813ef30 legacy=0x7f59d80af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:56:29.465 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 1 -- 172.21.15.112:0/3971234663 >> v1:172.21.15.112:6789/0 conn(0x7f59d80af8e0 legacy=0x7f59d8149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:29.465 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 10 monclient: _finish_hunting 0 2024-03-29T10:56:29.466 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 1 monclient: found mon.noname-b 2024-03-29T10:56:29.466 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:29.466 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:56:29.466 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 1 -- 172.21.15.112:0/3971234663 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d80ab1b0 con 0x7f59d80b0860 2024-03-29T10:56:29.466 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 10 monclient: _finish_auth 0 2024-03-29T10:56:29.466 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 10 monclient: _check_auth_tickets 2024-03-29T10:56:29.466 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:29.466 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 1 -- 172.21.15.112:0/3971234663 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f59d4001690 con 0x7f59d80b0860 2024-03-29T10:56:29.466 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:29.466 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:29.466 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 10 monclient: dump: 2024-03-29T10:56:29.467 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:29.467 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:29.467 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:29.467 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:29.467 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:29.467 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:29.467 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:29.467 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:29.467 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:29.467 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:29.467 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dfac1640 5 monclient: authenticate success, global_id 4424 2024-03-29T10:56:29.467 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dfac1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:29.468 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 1 -- 172.21.15.112:0/3971234663 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59d40017f0 con 0x7f59d80b0860 2024-03-29T10:56:29.468 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:29.468 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 1 -- 172.21.15.112:0/3971234663 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f59d4001c00 con 0x7f59d80b0860 2024-03-29T10:56:29.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:29.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:29.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.484+0000 7f59dc85b640 10 monclient: dump: 2024-03-29T10:56:29.469 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:29.469 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:29.469 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:29.469 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:29.469 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:29.469 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:29.469 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:29.469 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:29.469 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:29.469 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:29.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.485+0000 7f59dfac1640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:29.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.485+0000 7f59dfac1640 4 set_mon_vals no callback set 2024-03-29T10:56:29.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.485+0000 7f59dfac1640 10 monclient: shutdown 2024-03-29T10:56:29.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.485+0000 7f59dfac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:29.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.485+0000 7f59dfac1640 1 -- 172.21.15.112:0/3971234663 >> v1:172.21.15.112:6790/0 conn(0x7f59d80b0860 legacy=0x7f59d813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:29.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.486+0000 7f59dfac1640 1 -- 172.21.15.112:0/3971234663 shutdown_connections 2024-03-29T10:56:29.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.486+0000 7f59dfac1640 1 -- 172.21.15.112:0/3971234663 >> 172.21.15.112:0/3971234663 conn(0x7f59d806b660 msgr2=0x7f59d806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:29.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.486+0000 7f59dfac1640 1 -- 172.21.15.112:0/3971234663 shutdown_connections 2024-03-29T10:56:29.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.486+0000 7f59dfac1640 1 -- 172.21.15.112:0/3971234663 wait complete. 2024-03-29T10:56:29.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.487+0000 7f59dfac1640 10 monclient: build_initial_monmap 2024-03-29T10:56:29.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.487+0000 7f59dfac1640 10 monclient: monmap: 2024-03-29T10:56:29.470 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:56:29.470 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:29.470 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:56:29.471 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:56:29.471 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:29.471 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:29.471 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:56:29.471 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:56:29.471 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:56:29.471 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:29.471 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.488+0000 7f59dfac1640 1 librados: starting msgr at 2024-03-29T10:56:29.471 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.488+0000 7f59dfac1640 1 librados: starting objecter 2024-03-29T10:56:29.471 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.488+0000 7f59dfac1640 1 Processor -- start 2024-03-29T10:56:29.471 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.488+0000 7f59dfac1640 1 -- start start 2024-03-29T10:56:29.471 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.488+0000 7f59dfac1640 1 librados: setting wanted keys 2024-03-29T10:56:29.471 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.488+0000 7f59dfac1640 1 librados: calling monclient init 2024-03-29T10:56:29.472 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.488+0000 7f59dfac1640 10 monclient: init 2024-03-29T10:56:29.472 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.489+0000 7f59dfac1640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:29.472 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.489+0000 7f59dfac1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:56:29.475 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.492+0000 7f59dfac1640 10 monclient(hunting): picked mon.noname-b con 0x7f59d80af8e0 addr v1:172.21.15.118:6789/0 2024-03-29T10:56:29.475 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.492+0000 7f59dfac1640 10 monclient(hunting): picked mon.noname-a con 0x7f59d80b0860 addr v1:172.21.15.112:6789/0 2024-03-29T10:56:29.475 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.493+0000 7f59dd85d640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f59d80af8e0 0x7f59d81d66b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:35877/0 (socket says 172.21.15.112:35877) 2024-03-29T10:56:29.475 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.494+0000 7f59dfac1640 10 monclient(hunting): picked mon.noname-c con 0x7f59d813ef30 addr v1:172.21.15.112:6790/0 2024-03-29T10:56:29.476 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.494+0000 7f59dfac1640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d81d7370 con 0x7f59d80b0860 2024-03-29T10:56:29.476 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.494+0000 7f59dfac1640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d80b1610 con 0x7f59d80af8e0 2024-03-29T10:56:29.476 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.494+0000 7f59dfac1640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d80b41f0 con 0x7f59d813ef30 2024-03-29T10:56:29.476 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.494+0000 7f59dfac1640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:29.476 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.494+0000 7f59dfac1640 10 monclient(hunting): authenticate will time out at 1144.798474s 2024-03-29T10:56:29.476 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.494+0000 7f59dd85d640 1 -- 172.21.15.112:0/1678007718 learned_addr learned my addr 172.21.15.112:0/1678007718 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:56:29.477 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.494+0000 7f59ce7fc640 10 client.?.objecter ms_handle_connect 0x7f59d813ef30 2024-03-29T10:56:29.477 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.494+0000 7f59ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:29.477 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.494+0000 7f59ce7fc640 10 client.?.objecter ms_handle_connect 0x7f59d80af8e0 2024-03-29T10:56:29.477 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.494+0000 7f59ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:29.479 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.495+0000 7f59ce7fc640 1 -- 172.21.15.112:0/1678007718 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4241787671 0 0) 0x7f59d80b41f0 con 0x7f59d813ef30 2024-03-29T10:56:29.479 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.495+0000 7f59ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:29.479 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.495+0000 7f59ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:29.480 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.495+0000 7f59ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:29.480 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.495+0000 7f59ce7fc640 10 monclient(hunting): my global_id is 4427 2024-03-29T10:56:29.480 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.495+0000 7f59ce7fc640 1 -- 172.21.15.112:0/1678007718 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59bc004eb0 con 0x7f59d813ef30 2024-03-29T10:56:29.480 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.495+0000 7f59ce7fc640 1 -- 172.21.15.112:0/1678007718 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991791049 0 0) 0x7f59d80b1610 con 0x7f59d80af8e0 2024-03-29T10:56:29.480 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.495+0000 7f59ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:29.480 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.495+0000 7f59ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:29.480 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.495+0000 7f59ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:29.480 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.495+0000 7f59ce7fc640 10 monclient(hunting): my global_id is 4390 2024-03-29T10:56:29.480 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.495+0000 7f59ce7fc640 1 -- 172.21.15.112:0/1678007718 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59d80b41f0 con 0x7f59d80af8e0 2024-03-29T10:56:29.480 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.496+0000 7f59ce7fc640 1 -- 172.21.15.112:0/1678007718 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2471520493 0 0) 0x7f59bc004eb0 con 0x7f59d813ef30 2024-03-29T10:56:29.480 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.496+0000 7f59ce7fc640 1 -- 172.21.15.112:0/1678007718 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59d80b1610 con 0x7f59d813ef30 2024-03-29T10:56:29.480 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.497+0000 7f59ce7fc640 1 -- 172.21.15.112:0/1678007718 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f59c8001930 con 0x7f59d813ef30 2024-03-29T10:56:29.481 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.497+0000 7f59ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:29.481 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.497+0000 7f59ce7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:29.481 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.497+0000 7f59ce7fc640 10 monclient(hunting): dump: 2024-03-29T10:56:29.481 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:29.481 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:29.481 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:29.481 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:29.481 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:29.481 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:29.481 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:29.481 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:29.481 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:29.481 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:29.481 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.497+0000 7f59ce7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:29.481 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.497+0000 7f59ce7fc640 1 -- 172.21.15.112:0/1678007718 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 693419739 0 0) 0x7f59d80b1610 con 0x7f59d813ef30 2024-03-29T10:56:29.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.497+0000 7f59ce7fc640 1 -- 172.21.15.112:0/1678007718 >> v1:172.21.15.118:6789/0 conn(0x7f59d80af8e0 legacy=0x7f59d81d66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:29.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.497+0000 7f59ce7fc640 1 -- 172.21.15.112:0/1678007718 >> v1:172.21.15.112:6789/0 conn(0x7f59d80b0860 legacy=0x7f59d8136920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:29.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.497+0000 7f59ce7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:56:29.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.497+0000 7f59ce7fc640 1 monclient: found mon.c 2024-03-29T10:56:29.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.497+0000 7f59ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:29.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.497+0000 7f59ce7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:29.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.497+0000 7f59ce7fc640 1 -- 172.21.15.112:0/1678007718 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d81d6dc0 con 0x7f59d813ef30 2024-03-29T10:56:29.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.497+0000 7f59ce7fc640 10 monclient: _finish_auth 0 2024-03-29T10:56:29.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.497+0000 7f59ce7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:56:29.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.497+0000 7f59ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:29.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.497+0000 7f59dfac1640 5 monclient: authenticate success, global_id 4427 2024-03-29T10:56:29.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.497+0000 7f59dfac1640 10 monclient: _renew_subs 2024-03-29T10:56:29.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.497+0000 7f59dfac1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:29.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.497+0000 7f59dfac1640 1 -- 172.21.15.112:0/1678007718 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d81d7070 con 0x7f59d813ef30 2024-03-29T10:56:29.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.498+0000 7f59dfac1640 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:29.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.498+0000 7f59dfac1640 10 monclient: _renew_subs 2024-03-29T10:56:29.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.498+0000 7f59dfac1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:29.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.498+0000 7f59dfac1640 1 -- 172.21.15.112:0/1678007718 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d81deb70 con 0x7f59d813ef30 2024-03-29T10:56:29.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.498+0000 7f59dfac1640 1 librados: init done 2024-03-29T10:56:29.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.498+0000 7f59ce7fc640 1 -- 172.21.15.112:0/1678007718 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59c8001a90 con 0x7f59d813ef30 2024-03-29T10:56:29.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.498+0000 7f59ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:29.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.498+0000 7f59cffff640 4 set_mon_vals no callback set 2024-03-29T10:56:29.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.499+0000 7f59ce7fc640 1 -- 172.21.15.112:0/1678007718 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f59c80037f0 con 0x7f59d813ef30 2024-03-29T10:56:29.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.499+0000 7f59ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:29.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.499+0000 7f59ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:29.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.499+0000 7f59ce7fc640 10 monclient: dump: 2024-03-29T10:56:29.486 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:29.486 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:29.486 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:29.486 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:29.486 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:29.487 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:29.487 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:29.487 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:29.487 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:29.487 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:29.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.500+0000 7f59dfac1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:29.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.500+0000 7f59dfac1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:29.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.500+0000 7f59dfac1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:29.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.500+0000 7f59dfac1640 1 -- 172.21.15.112:0/1678007718 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c0005080 con 0x7f59d813ef30 2024-03-29T10:56:29.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.500+0000 7f59ce7fc640 1 -- 172.21.15.112:0/1678007718 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f59c8003a10 con 0x7f59d813ef30 2024-03-29T10:56:29.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.501+0000 7f59dd85d640 10 monclient: get_auth_request con 0x7f59bc06dd90 auth_method 0 2024-03-29T10:56:29.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.501+0000 7f59ce7fc640 1 -- 172.21.15.112:0/1678007718 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f59c80838a0 con 0x7f59d813ef30 2024-03-29T10:56:29.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.501+0000 7f59ce7fc640 10 client.4427.objecter ms_dispatch 0x7f59d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:29.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.501+0000 7f59ce7fc640 3 client.4427.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:29.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.501+0000 7f59ce7fc640 3 client.4427.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:29.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.501+0000 7f59ce7fc640 20 client.4427.objecter dump_active .. 0 homeless 2024-03-29T10:56:29.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.501+0000 7f59ce7fc640 10 client.4427.objecter ms_handle_connect 0x7f59bc06dd90 2024-03-29T10:56:29.490 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.508+0000 7f59ce7fc640 1 -- 172.21.15.112:0/1678007718 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f59c804f180 con 0x7f59d813ef30 2024-03-29T10:56:29.490 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.508+0000 7f59ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:29.490 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.508+0000 7f59ce7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:29.849 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.868+0000 7f59dfac1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:29.849 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.868+0000 7f59dfac1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:29.849 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.868+0000 7f59dfac1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:29.849 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.868+0000 7f59dfac1640 1 -- 172.21.15.112:0/1678007718 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59c0005330 con 0x7f59d813ef30 2024-03-29T10:56:29.851 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.870+0000 7f59ce7fc640 1 -- 172.21.15.112:0/1678007718 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f59c80510b0 con 0x7f59d813ef30 2024-03-29T10:56:29.851 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.870+0000 7f59ce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:29.851 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.870+0000 7f59ce7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:29.852 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:56:29.852 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:29.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.874+0000 7f59dfac1640 10 librados: watch_flush enter 2024-03-29T10:56:29.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.874+0000 7f59dfac1640 10 librados: watch_flush exit 2024-03-29T10:56:29.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.874+0000 7f59dfac1640 20 client.4427.objecter shutdown clearing up homeless session... 2024-03-29T10:56:29.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.874+0000 7f59dfac1640 10 client.4427.objecter successfully canceled tick 2024-03-29T10:56:29.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.874+0000 7f59dfac1640 1 -- 172.21.15.112:0/1678007718 >> v1:172.21.15.118:6816/3181073408 conn(0x7f59bc06dd90 legacy=0x7f59bc070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:29.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.874+0000 7f59dfac1640 10 monclient: shutdown 2024-03-29T10:56:29.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.875+0000 7f59dfac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:29.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.875+0000 7f59dfac1640 1 -- 172.21.15.112:0/1678007718 >> v1:172.21.15.112:6790/0 conn(0x7f59d813ef30 legacy=0x7f59d8137030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:29.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.875+0000 7f59dfac1640 1 -- 172.21.15.112:0/1678007718 shutdown_connections 2024-03-29T10:56:29.857 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.875+0000 7f59dfac1640 1 -- 172.21.15.112:0/1678007718 >> 172.21.15.112:0/1678007718 conn(0x7f59d806b660 msgr2=0x7f59d813e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:29.857 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.875+0000 7f59dfac1640 1 -- 172.21.15.112:0/1678007718 shutdown_connections 2024-03-29T10:56:29.857 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.875+0000 7f59dfac1640 1 -- 172.21.15.112:0/1678007718 wait complete. 2024-03-29T10:56:29.857 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:29.875+0000 7f59dfac1640 1 librados: shutdown 2024-03-29T10:56:29.872 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:36.682 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:36.846 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.865+0000 7f1fda7d7640 10 monclient: get_monmap_and_config 2024-03-29T10:56:36.846 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.865+0000 7f1fda7d7640 10 monclient: build_initial_monmap 2024-03-29T10:56:36.846 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.865+0000 7f1fda7d7640 10 monclient: monmap: 2024-03-29T10:56:36.846 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:56:36.846 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:36.846 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:56:36.866641+0000 2024-03-29T10:56:36.847 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:56:36.866641+0000 2024-03-29T10:56:36.847 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:36.847 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:36.847 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:56:36.847 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:56:36.847 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:56:36.847 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:36.848 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.866+0000 7f1fda7d7640 1 Processor -- start 2024-03-29T10:56:36.848 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.866+0000 7f1fda7d7640 1 -- start start 2024-03-29T10:56:36.848 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.866+0000 7f1fda7d7640 10 monclient: init 2024-03-29T10:56:36.848 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.867+0000 7f1fda7d7640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:36.848 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.867+0000 7f1fda7d7640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:36.850 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.867+0000 7f1fda7d7640 10 monclient(hunting): picked mon.noname-b con 0x7f1fd41422b0 addr v1:172.21.15.112:6790/0 2024-03-29T10:56:36.851 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.870+0000 7f1fda7d7640 10 monclient(hunting): picked mon.noname-c con 0x7f1fd4142690 addr v1:172.21.15.118:6789/0 2024-03-29T10:56:36.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.870+0000 7f1fda7d7640 10 monclient(hunting): picked mon.noname-a con 0x7f1fd40b17d0 addr v1:172.21.15.112:6789/0 2024-03-29T10:56:36.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.871+0000 7f1fda7d7640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd4142c80 con 0x7f1fd40b17d0 2024-03-29T10:56:36.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.871+0000 7f1fd3fff640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f1fd41422b0 0x7f1fd413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:34893/0 (socket says 172.21.15.112:34893) 2024-03-29T10:56:36.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.871+0000 7f1fd3fff640 1 -- 172.21.15.112:0/3601238403 learned_addr learned my addr 172.21.15.112:0/3601238403 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:56:36.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.871+0000 7f1fda7d7640 1 -- 172.21.15.112:0/3601238403 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd40aae40 con 0x7f1fd4142690 2024-03-29T10:56:36.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.871+0000 7f1fda7d7640 1 -- 172.21.15.112:0/3601238403 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd413fbd0 con 0x7f1fd41422b0 2024-03-29T10:56:36.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.871+0000 7f1fda7d7640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:36.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.871+0000 7f1fda7d7640 10 monclient(hunting): authenticate will time out at 1152.175706s 2024-03-29T10:56:36.854 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.871+0000 7f1fd2ffd640 1 -- 172.21.15.112:0/3601238403 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4036399861 0 0) 0x7f1fd413fbd0 con 0x7f1fd41422b0 2024-03-29T10:56:36.854 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.872+0000 7f1fd2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:36.854 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.872+0000 7f1fd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:36.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.872+0000 7f1fd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:36.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.872+0000 7f1fd2ffd640 10 monclient(hunting): my global_id is 4430 2024-03-29T10:56:36.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.872+0000 7f1fd2ffd640 1 -- 172.21.15.112:0/3601238403 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fbc003520 con 0x7f1fd41422b0 2024-03-29T10:56:36.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.872+0000 7f1fd2ffd640 1 -- 172.21.15.112:0/3601238403 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3589851070 0 0) 0x7f1fbc003520 con 0x7f1fd41422b0 2024-03-29T10:56:36.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.872+0000 7f1fd2ffd640 1 -- 172.21.15.112:0/3601238403 >> v1:172.21.15.118:6789/0 conn(0x7f1fd4142690 legacy=0x7f1fd40b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:36.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.872+0000 7f1fd2ffd640 1 -- 172.21.15.112:0/3601238403 >> v1:172.21.15.112:6789/0 conn(0x7f1fd40b17d0 legacy=0x7f1fd40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:56:36.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.872+0000 7f1fd2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:36.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.872+0000 7f1fd2ffd640 1 monclient: found mon.noname-b 2024-03-29T10:56:36.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.872+0000 7f1fd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:36.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.872+0000 7f1fd2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:56:36.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.872+0000 7f1fd2ffd640 1 -- 172.21.15.112:0/3601238403 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd4151940 con 0x7f1fd41422b0 2024-03-29T10:56:36.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.872+0000 7f1fd2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:56:36.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.872+0000 7f1fd2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:36.855 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.872+0000 7f1fd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:36.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.872+0000 7f1fd2ffd640 1 -- 172.21.15.112:0/3601238403 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1fc4001690 con 0x7f1fd41422b0 2024-03-29T10:56:36.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.872+0000 7f1fd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:36.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.873+0000 7f1fd2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:36.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.873+0000 7f1fd2ffd640 10 monclient: dump: 2024-03-29T10:56:36.856 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:36.856 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:36.856 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:36.856 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:36.856 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:36.856 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:36.856 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:36.856 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:36.856 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:36.856 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:36.856 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.873+0000 7f1fda7d7640 5 monclient: authenticate success, global_id 4430 2024-03-29T10:56:36.857 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.873+0000 7f1fda7d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:36.857 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.873+0000 7f1fd2ffd640 1 -- 172.21.15.112:0/3601238403 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fc40018f0 con 0x7f1fd41422b0 2024-03-29T10:56:36.857 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.873+0000 7f1fd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:36.857 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.873+0000 7f1fda7d7640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:36.857 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.873+0000 7f1fda7d7640 4 set_mon_vals no callback set 2024-03-29T10:56:36.857 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.874+0000 7f1fd2ffd640 1 -- 172.21.15.112:0/3601238403 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1fc4001cc0 con 0x7f1fd41422b0 2024-03-29T10:56:36.857 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.874+0000 7f1fd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:36.857 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.874+0000 7f1fd2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:36.857 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.874+0000 7f1fd2ffd640 10 monclient: dump: 2024-03-29T10:56:36.857 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:36.857 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:36.857 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:36.857 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:36.857 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:36.857 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:36.858 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:36.858 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:36.858 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:36.858 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:36.858 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.874+0000 7f1fda7d7640 10 monclient: shutdown 2024-03-29T10:56:36.858 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.874+0000 7f1fda7d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:36.858 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.874+0000 7f1fda7d7640 1 -- 172.21.15.112:0/3601238403 >> v1:172.21.15.112:6790/0 conn(0x7f1fd41422b0 legacy=0x7f1fd413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:36.858 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.876+0000 7f1fda7d7640 1 -- 172.21.15.112:0/3601238403 shutdown_connections 2024-03-29T10:56:36.859 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.876+0000 7f1fda7d7640 1 -- 172.21.15.112:0/3601238403 >> 172.21.15.112:0/3601238403 conn(0x7f1fd4066a00 msgr2=0x7f1fd4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:36.860 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.878+0000 7f1fda7d7640 1 -- 172.21.15.112:0/3601238403 shutdown_connections 2024-03-29T10:56:36.861 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.881+0000 7f1fda7d7640 1 -- 172.21.15.112:0/3601238403 wait complete. 2024-03-29T10:56:36.863 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.882+0000 7f1fda7d7640 10 monclient: build_initial_monmap 2024-03-29T10:56:36.863 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.882+0000 7f1fda7d7640 10 monclient: monmap: 2024-03-29T10:56:36.863 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:56:36.863 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:36.863 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:56:36.863 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:56:36.863 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:36.863 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:36.863 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:56:36.863 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:56:36.864 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:56:36.864 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:36.867 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.883+0000 7f1fda7d7640 1 librados: starting msgr at 2024-03-29T10:56:36.867 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.883+0000 7f1fda7d7640 1 librados: starting objecter 2024-03-29T10:56:36.867 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.883+0000 7f1fda7d7640 1 Processor -- start 2024-03-29T10:56:36.867 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.884+0000 7f1fda7d7640 1 -- start start 2024-03-29T10:56:36.867 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.884+0000 7f1fda7d7640 1 librados: setting wanted keys 2024-03-29T10:56:36.868 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.884+0000 7f1fda7d7640 1 librados: calling monclient init 2024-03-29T10:56:36.868 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.884+0000 7f1fda7d7640 10 monclient: init 2024-03-29T10:56:36.868 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.884+0000 7f1fda7d7640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:36.868 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.884+0000 7f1fda7d7640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:56:36.868 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.884+0000 7f1fda7d7640 10 monclient(hunting): picked mon.noname-c con 0x7f1fd40b17d0 addr v1:172.21.15.112:6790/0 2024-03-29T10:56:36.868 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.884+0000 7f1fda7d7640 10 monclient(hunting): picked mon.noname-a con 0x7f1fd41422b0 addr v1:172.21.15.112:6789/0 2024-03-29T10:56:36.868 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.884+0000 7f1fda7d7640 10 monclient(hunting): picked mon.noname-b con 0x7f1fd4142690 addr v1:172.21.15.118:6789/0 2024-03-29T10:56:36.868 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.884+0000 7f1fda7d7640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd40b71a0 con 0x7f1fd41422b0 2024-03-29T10:56:36.868 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.884+0000 7f1fda7d7640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd40ac840 con 0x7f1fd4142690 2024-03-29T10:56:36.868 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.884+0000 7f1fda7d7640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd40acd90 con 0x7f1fd40b17d0 2024-03-29T10:56:36.868 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.884+0000 7f1fda7d7640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:36.868 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.884+0000 7f1fda7d7640 10 monclient(hunting): authenticate will time out at 1152.189118s 2024-03-29T10:56:36.868 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.885+0000 7f1fd37fe640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f1fd41422b0 0x7f1fd412cb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60306/0 (socket says 172.21.15.112:60306) 2024-03-29T10:56:36.868 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.885+0000 7f1fd37fe640 1 -- 172.21.15.112:0/3244213552 learned_addr learned my addr 172.21.15.112:0/3244213552 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:56:36.869 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.887+0000 7f1fd0ff9640 10 client.?.objecter ms_handle_connect 0x7f1fd41422b0 2024-03-29T10:56:36.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.887+0000 7f1fd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:36.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.887+0000 7f1fd0ff9640 10 client.?.objecter ms_handle_connect 0x7f1fd4142690 2024-03-29T10:56:36.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.887+0000 7f1fd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:36.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.887+0000 7f1fd0ff9640 1 -- 172.21.15.112:0/3244213552 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4279701648 0 0) 0x7f1fd40ac840 con 0x7f1fd4142690 2024-03-29T10:56:36.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.887+0000 7f1fd0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:36.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.887+0000 7f1fd0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:36.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.887+0000 7f1fd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:36.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.887+0000 7f1fd0ff9640 10 monclient(hunting): my global_id is 4393 2024-03-29T10:56:36.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.887+0000 7f1fd0ff9640 1 -- 172.21.15.112:0/3244213552 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fc0004eb0 con 0x7f1fd4142690 2024-03-29T10:56:36.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.888+0000 7f1fd0ff9640 1 -- 172.21.15.112:0/3244213552 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2448444021 0 0) 0x7f1fc0004eb0 con 0x7f1fd4142690 2024-03-29T10:56:36.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.888+0000 7f1fd0ff9640 1 -- 172.21.15.112:0/3244213552 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fd40ac840 con 0x7f1fd4142690 2024-03-29T10:56:36.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.888+0000 7f1fd0ff9640 1 -- 172.21.15.112:0/3244213552 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1fcc001aa0 con 0x7f1fd4142690 2024-03-29T10:56:36.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.888+0000 7f1fd0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:36.871 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.888+0000 7f1fd0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:36.871 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.888+0000 7f1fd0ff9640 10 monclient(hunting): dump: 2024-03-29T10:56:36.871 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:36.871 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:36.871 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:36.871 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:36.871 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:36.871 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:36.871 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:36.871 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:36.871 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:36.871 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:36.871 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.888+0000 7f1fd0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:36.871 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.888+0000 7f1fd0ff9640 1 -- 172.21.15.112:0/3244213552 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2064405886 0 0) 0x7f1fd40ac840 con 0x7f1fd4142690 2024-03-29T10:56:36.871 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.888+0000 7f1fd0ff9640 1 -- 172.21.15.112:0/3244213552 >> v1:172.21.15.112:6790/0 conn(0x7f1fd40b17d0 legacy=0x7f1fd40b5780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:36.874 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.889+0000 7f1fd0ff9640 1 -- 172.21.15.112:0/3244213552 >> v1:172.21.15.112:6789/0 conn(0x7f1fd41422b0 legacy=0x7f1fd412cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:36.874 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.890+0000 7f1fd0ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:56:36.874 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.890+0000 7f1fd0ff9640 1 monclient: found mon.b 2024-03-29T10:56:36.874 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.890+0000 7f1fd0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:36.874 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.890+0000 7f1fd0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:56:36.874 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.890+0000 7f1fd0ff9640 1 -- 172.21.15.112:0/3244213552 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd412d990 con 0x7f1fd4142690 2024-03-29T10:56:36.874 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.890+0000 7f1fd0ff9640 10 monclient: _finish_auth 0 2024-03-29T10:56:36.874 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.890+0000 7f1fd0ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:56:36.874 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.890+0000 7f1fd0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:36.874 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.890+0000 7f1fda7d7640 5 monclient: authenticate success, global_id 4393 2024-03-29T10:56:36.874 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.890+0000 7f1fda7d7640 10 monclient: _renew_subs 2024-03-29T10:56:36.875 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.890+0000 7f1fda7d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:56:36.875 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.890+0000 7f1fda7d7640 1 -- 172.21.15.112:0/3244213552 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd40acab0 con 0x7f1fd4142690 2024-03-29T10:56:36.875 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.890+0000 7f1fda7d7640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:36.875 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.890+0000 7f1fda7d7640 10 monclient: _renew_subs 2024-03-29T10:56:36.875 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.890+0000 7f1fda7d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:56:36.875 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.890+0000 7f1fda7d7640 1 -- 172.21.15.112:0/3244213552 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd40b7820 con 0x7f1fd4142690 2024-03-29T10:56:36.875 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.890+0000 7f1fda7d7640 1 librados: init done 2024-03-29T10:56:36.875 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.890+0000 7f1fd0ff9640 1 -- 172.21.15.112:0/3244213552 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fcc002030 con 0x7f1fd4142690 2024-03-29T10:56:36.875 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.890+0000 7f1fd0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:36.875 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.890+0000 7f1fd1ffb640 4 set_mon_vals no callback set 2024-03-29T10:56:36.875 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.891+0000 7f1fd0ff9640 1 -- 172.21.15.112:0/3244213552 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1fcc003b10 con 0x7f1fd4142690 2024-03-29T10:56:36.875 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.892+0000 7f1fd0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:36.875 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.892+0000 7f1fd0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:36.875 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.892+0000 7f1fd0ff9640 10 monclient: dump: 2024-03-29T10:56:36.875 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:36.876 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:36.876 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:36.876 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:36.876 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:36.876 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:36.876 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:36.876 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:36.876 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:36.876 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:36.876 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.892+0000 7f1fda7d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:36.876 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.892+0000 7f1fda7d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:36.876 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.892+0000 7f1fda7d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:56:36.876 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.892+0000 7f1fda7d7640 1 -- 172.21.15.112:0/3244213552 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fb4005080 con 0x7f1fd4142690 2024-03-29T10:56:36.876 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.893+0000 7f1fd0ff9640 1 -- 172.21.15.112:0/3244213552 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f1fcc003c70 con 0x7f1fd4142690 2024-03-29T10:56:36.877 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.894+0000 7f1fd0ff9640 1 -- 172.21.15.112:0/3244213552 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f1fcc083ab0 con 0x7f1fd4142690 2024-03-29T10:56:36.878 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.894+0000 7f1fd0ff9640 10 client.4393.objecter ms_dispatch 0x7f1fd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:36.878 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.894+0000 7f1fd0ff9640 3 client.4393.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:36.878 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.894+0000 7f1fd3fff640 10 monclient: get_auth_request con 0x7f1fc006dbe0 auth_method 0 2024-03-29T10:56:36.878 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.894+0000 7f1fd0ff9640 3 client.4393.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:36.878 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.894+0000 7f1fd0ff9640 20 client.4393.objecter dump_active .. 0 homeless 2024-03-29T10:56:36.878 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.894+0000 7f1fd0ff9640 10 client.4393.objecter ms_handle_connect 0x7f1fc006dbe0 2024-03-29T10:56:36.882 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.901+0000 7f1fd0ff9640 1 -- 172.21.15.112:0/3244213552 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1fcc04f3b0 con 0x7f1fd4142690 2024-03-29T10:56:36.882 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.901+0000 7f1fd0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:36.882 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:36.901+0000 7f1fd0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:37.234 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:37.252+0000 7f1fda7d7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:37.234 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:37.252+0000 7f1fda7d7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:37.234 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:37.252+0000 7f1fda7d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:56:37.235 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:37.252+0000 7f1fda7d7640 1 -- 172.21.15.112:0/3244213552 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fb4005330 con 0x7f1fd4142690 2024-03-29T10:56:37.237 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:37.255+0000 7f1fd0ff9640 1 -- 172.21.15.112:0/3244213552 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f1fcc0512e0 con 0x7f1fd4142690 2024-03-29T10:56:37.237 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:37.255+0000 7f1fd0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:37.237 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:37.255+0000 7f1fd0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:37.238 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:56:37.238 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:37.242 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:37.260+0000 7f1fda7d7640 10 librados: watch_flush enter 2024-03-29T10:56:37.242 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:37.260+0000 7f1fda7d7640 10 librados: watch_flush exit 2024-03-29T10:56:37.242 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:37.260+0000 7f1fda7d7640 20 client.4393.objecter shutdown clearing up homeless session... 2024-03-29T10:56:37.242 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:37.260+0000 7f1fda7d7640 10 client.4393.objecter successfully canceled tick 2024-03-29T10:56:37.242 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:37.260+0000 7f1fda7d7640 1 -- 172.21.15.112:0/3244213552 >> v1:172.21.15.118:6816/3181073408 conn(0x7f1fc006dbe0 legacy=0x7f1fc00700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:37.242 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:37.260+0000 7f1fda7d7640 10 monclient: shutdown 2024-03-29T10:56:37.242 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:37.260+0000 7f1fda7d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:37.242 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:37.260+0000 7f1fda7d7640 1 -- 172.21.15.112:0/3244213552 >> v1:172.21.15.118:6789/0 conn(0x7f1fd4142690 legacy=0x7f1fd412d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:37.242 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:37.260+0000 7f1fda7d7640 1 -- 172.21.15.112:0/3244213552 shutdown_connections 2024-03-29T10:56:37.242 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:37.260+0000 7f1fda7d7640 1 -- 172.21.15.112:0/3244213552 >> 172.21.15.112:0/3244213552 conn(0x7f1fd4066a00 msgr2=0x7f1fd40b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:37.243 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:37.261+0000 7f1fda7d7640 1 -- 172.21.15.112:0/3244213552 shutdown_connections 2024-03-29T10:56:37.243 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:37.261+0000 7f1fda7d7640 1 -- 172.21.15.112:0/3244213552 wait complete. 2024-03-29T10:56:37.243 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:37.261+0000 7f1fda7d7640 1 librados: shutdown 2024-03-29T10:56:37.262 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:44.070 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:44.213 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.232+0000 7fd063304640 10 monclient: get_monmap_and_config 2024-03-29T10:56:44.213 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.232+0000 7fd063304640 10 monclient: build_initial_monmap 2024-03-29T10:56:44.213 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.232+0000 7fd063304640 10 monclient: monmap: 2024-03-29T10:56:44.213 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:56:44.213 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:44.213 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:56:44.233137+0000 2024-03-29T10:56:44.213 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:56:44.233137+0000 2024-03-29T10:56:44.213 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:44.213 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:44.213 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:56:44.213 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:56:44.214 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:56:44.214 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:44.214 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.233+0000 7fd063304640 1 Processor -- start 2024-03-29T10:56:44.214 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.233+0000 7fd063304640 1 -- start start 2024-03-29T10:56:44.214 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.233+0000 7fd063304640 10 monclient: init 2024-03-29T10:56:44.217 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.234+0000 7fd063304640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:44.217 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.234+0000 7fd063304640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:56:44.217 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.234+0000 7fd063304640 10 monclient(hunting): picked mon.noname-a con 0x7fd05c0b1f60 addr v1:172.21.15.112:6789/0 2024-03-29T10:56:44.217 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.234+0000 7fd063304640 10 monclient(hunting): picked mon.noname-c con 0x7fd05c071eb0 addr v1:172.21.15.118:6789/0 2024-03-29T10:56:44.218 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.234+0000 7fd063304640 10 monclient(hunting): picked mon.noname-b con 0x7fd05c143d10 addr v1:172.21.15.112:6790/0 2024-03-29T10:56:44.218 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.234+0000 7fd063304640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd05c06ef80 con 0x7fd05c0b1f60 2024-03-29T10:56:44.218 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.234+0000 7fd063304640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd05c06f260 con 0x7fd05c071eb0 2024-03-29T10:56:44.218 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.234+0000 7fd063304640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd05c0b40c0 con 0x7fd05c143d10 2024-03-29T10:56:44.218 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.234+0000 7fd063304640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:44.218 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.234+0000 7fd063304640 10 monclient(hunting): authenticate will time out at 1159.538838s 2024-03-29T10:56:44.218 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.234+0000 7fd0610a0640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fd05c0b1f60 0x7fd05c143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46820/0 (socket says 172.21.15.112:46820) 2024-03-29T10:56:44.218 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.234+0000 7fd0610a0640 1 -- 172.21.15.112:0/3120663507 learned_addr learned my addr 172.21.15.112:0/3120663507 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:56:44.218 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 1 -- 172.21.15.112:0/3120663507 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606030164 0 0) 0x7fd05c06f260 con 0x7fd05c071eb0 2024-03-29T10:56:44.218 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 10 monclient(hunting): _negotiate 2024-03-29T10:56:44.218 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:44.218 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:44.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 10 monclient(hunting): my global_id is 4396 2024-03-29T10:56:44.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 1 -- 172.21.15.112:0/3120663507 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd048003520 con 0x7fd05c071eb0 2024-03-29T10:56:44.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 1 -- 172.21.15.112:0/3120663507 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233926005 0 0) 0x7fd05c06ef80 con 0x7fd05c0b1f60 2024-03-29T10:56:44.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 10 monclient(hunting): _negotiate 2024-03-29T10:56:44.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:44.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:44.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 10 monclient(hunting): my global_id is 4407 2024-03-29T10:56:44.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 1 -- 172.21.15.112:0/3120663507 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd05c06f260 con 0x7fd05c0b1f60 2024-03-29T10:56:44.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 1 -- 172.21.15.112:0/3120663507 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285541750 0 0) 0x7fd05c0b40c0 con 0x7fd05c143d10 2024-03-29T10:56:44.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 10 monclient(hunting): _negotiate 2024-03-29T10:56:44.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:44.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:44.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 10 monclient(hunting): my global_id is 4433 2024-03-29T10:56:44.220 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 1 -- 172.21.15.112:0/3120663507 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd05c06ef80 con 0x7fd05c143d10 2024-03-29T10:56:44.220 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 1 -- 172.21.15.112:0/3120663507 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4103970252 0 0) 0x7fd05c06f260 con 0x7fd05c0b1f60 2024-03-29T10:56:44.220 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 1 -- 172.21.15.112:0/3120663507 >> v1:172.21.15.112:6790/0 conn(0x7fd05c143d10 legacy=0x7fd05c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:44.220 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 1 -- 172.21.15.112:0/3120663507 >> v1:172.21.15.118:6789/0 conn(0x7fd05c071eb0 legacy=0x7fd05c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:44.220 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 10 monclient: _finish_hunting 0 2024-03-29T10:56:44.220 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 1 monclient: found mon.noname-a 2024-03-29T10:56:44.220 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:44.220 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T10:56:44.220 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 1 -- 172.21.15.112:0/3120663507 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05c155d70 con 0x7fd05c0b1f60 2024-03-29T10:56:44.220 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 10 monclient: _finish_auth 0 2024-03-29T10:56:44.220 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 10 monclient: _check_auth_tickets 2024-03-29T10:56:44.220 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd053fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:44.220 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd063304640 5 monclient: authenticate success, global_id 4407 2024-03-29T10:56:44.221 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.235+0000 7fd063304640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:44.221 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.236+0000 7fd053fff640 1 -- 172.21.15.112:0/3120663507 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fd058001560 con 0x7fd05c0b1f60 2024-03-29T10:56:44.221 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.236+0000 7fd053fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:44.221 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.236+0000 7fd053fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:56:44.221 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.236+0000 7fd053fff640 10 monclient: dump: 2024-03-29T10:56:44.221 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:44.221 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:44.221 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:44.221 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:44.221 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:44.221 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:44.222 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:44.222 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:44.222 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:44.222 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:44.222 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.236+0000 7fd063304640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:44.222 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.236+0000 7fd053fff640 1 -- 172.21.15.112:0/3120663507 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd0580017b0 con 0x7fd05c0b1f60 2024-03-29T10:56:44.222 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.236+0000 7fd053fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:44.222 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.236+0000 7fd053fff640 1 -- 172.21.15.112:0/3120663507 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fd058001b40 con 0x7fd05c0b1f60 2024-03-29T10:56:44.222 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.236+0000 7fd063304640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:44.222 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.236+0000 7fd063304640 4 set_mon_vals no callback set 2024-03-29T10:56:44.222 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.236+0000 7fd053fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:44.223 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.236+0000 7fd053fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:44.223 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.236+0000 7fd053fff640 10 monclient: dump: 2024-03-29T10:56:44.223 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:44.223 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:44.223 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:44.223 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:44.223 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:44.223 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:44.223 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:44.223 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:44.223 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:44.223 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:44.223 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.237+0000 7fd063304640 10 monclient: shutdown 2024-03-29T10:56:44.224 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.237+0000 7fd063304640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:44.224 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.237+0000 7fd063304640 1 -- 172.21.15.112:0/3120663507 >> v1:172.21.15.112:6789/0 conn(0x7fd05c0b1f60 legacy=0x7fd05c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:44.224 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.237+0000 7fd063304640 1 -- 172.21.15.112:0/3120663507 shutdown_connections 2024-03-29T10:56:44.227 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.237+0000 7fd063304640 1 -- 172.21.15.112:0/3120663507 >> 172.21.15.112:0/3120663507 conn(0x7fd05c066a00 msgr2=0x7fd05c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:44.227 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.237+0000 7fd063304640 1 -- 172.21.15.112:0/3120663507 shutdown_connections 2024-03-29T10:56:44.227 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.237+0000 7fd063304640 1 -- 172.21.15.112:0/3120663507 wait complete. 2024-03-29T10:56:44.228 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.238+0000 7fd063304640 10 monclient: build_initial_monmap 2024-03-29T10:56:44.228 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.238+0000 7fd063304640 10 monclient: monmap: 2024-03-29T10:56:44.228 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:56:44.228 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:44.228 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:56:44.228 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:56:44.228 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:44.228 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:44.228 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:56:44.228 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:56:44.229 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:56:44.229 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:44.229 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.238+0000 7fd063304640 1 librados: starting msgr at 2024-03-29T10:56:44.229 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.238+0000 7fd063304640 1 librados: starting objecter 2024-03-29T10:56:44.229 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.239+0000 7fd063304640 1 Processor -- start 2024-03-29T10:56:44.229 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.239+0000 7fd063304640 1 -- start start 2024-03-29T10:56:44.229 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.239+0000 7fd063304640 1 librados: setting wanted keys 2024-03-29T10:56:44.229 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.239+0000 7fd063304640 1 librados: calling monclient init 2024-03-29T10:56:44.229 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.239+0000 7fd063304640 10 monclient: init 2024-03-29T10:56:44.229 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.239+0000 7fd063304640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:44.229 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.239+0000 7fd063304640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:56:44.229 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.239+0000 7fd063304640 10 monclient(hunting): picked mon.noname-b con 0x7fd05c0b1f60 addr v1:172.21.15.118:6789/0 2024-03-29T10:56:44.230 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.240+0000 7fd063304640 10 monclient(hunting): picked mon.noname-c con 0x7fd05c143d10 addr v1:172.21.15.112:6790/0 2024-03-29T10:56:44.230 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.240+0000 7fd063304640 10 monclient(hunting): picked mon.noname-a con 0x7fd05c154750 addr v1:172.21.15.112:6789/0 2024-03-29T10:56:44.230 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.240+0000 7fd063304640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd05c1402d0 con 0x7fd05c154750 2024-03-29T10:56:44.230 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.240+0000 7fd063304640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd05c14ad70 con 0x7fd05c0b1f60 2024-03-29T10:56:44.230 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.240+0000 7fd063304640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd05c14b050 con 0x7fd05c143d10 2024-03-29T10:56:44.230 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.240+0000 7fd063304640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:44.230 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.240+0000 7fd063304640 10 monclient(hunting): authenticate will time out at 1159.544261s 2024-03-29T10:56:44.230 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.240+0000 7fd0610a0640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fd05c0b1f60 0x7fd05c13f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:52002/0 (socket says 172.21.15.112:52002) 2024-03-29T10:56:44.230 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.240+0000 7fd0610a0640 1 -- 172.21.15.112:0/3391590031 learned_addr learned my addr 172.21.15.112:0/3391590031 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:56:44.230 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.240+0000 7fd051ffb640 10 client.?.objecter ms_handle_connect 0x7fd05c154750 2024-03-29T10:56:44.231 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.240+0000 7fd051ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:44.231 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.240+0000 7fd051ffb640 10 client.?.objecter ms_handle_connect 0x7fd05c143d10 2024-03-29T10:56:44.231 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.240+0000 7fd051ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:44.231 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.240+0000 7fd051ffb640 1 -- 172.21.15.112:0/3391590031 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2430383274 0 0) 0x7fd05c14b050 con 0x7fd05c143d10 2024-03-29T10:56:44.231 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.240+0000 7fd051ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:56:44.231 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.240+0000 7fd051ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:44.231 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.240+0000 7fd051ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:44.231 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.240+0000 7fd051ffb640 10 monclient(hunting): my global_id is 4436 2024-03-29T10:56:44.231 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.240+0000 7fd051ffb640 1 -- 172.21.15.112:0/3391590031 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd040004eb0 con 0x7fd05c143d10 2024-03-29T10:56:44.231 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.240+0000 7fd051ffb640 1 -- 172.21.15.112:0/3391590031 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395013435 0 0) 0x7fd05c1402d0 con 0x7fd05c154750 2024-03-29T10:56:44.232 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.241+0000 7fd051ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:56:44.232 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.241+0000 7fd051ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:44.232 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.241+0000 7fd051ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:44.232 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.241+0000 7fd051ffb640 10 monclient(hunting): my global_id is 4410 2024-03-29T10:56:44.232 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.241+0000 7fd051ffb640 1 -- 172.21.15.112:0/3391590031 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd05c14b050 con 0x7fd05c154750 2024-03-29T10:56:44.232 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.241+0000 7fd051ffb640 1 -- 172.21.15.112:0/3391590031 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4289373131 0 0) 0x7fd040004eb0 con 0x7fd05c143d10 2024-03-29T10:56:44.232 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.241+0000 7fd051ffb640 1 -- 172.21.15.112:0/3391590031 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd05c1402d0 con 0x7fd05c143d10 2024-03-29T10:56:44.232 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.241+0000 7fd051ffb640 1 -- 172.21.15.112:0/3391590031 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fd04c001950 con 0x7fd05c143d10 2024-03-29T10:56:44.232 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.241+0000 7fd051ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:44.232 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.241+0000 7fd051ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:44.232 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.241+0000 7fd051ffb640 10 monclient(hunting): dump: 2024-03-29T10:56:44.232 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:44.232 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:44.233 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:44.233 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:44.233 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:44.233 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:44.233 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:44.233 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:44.233 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:44.233 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:44.233 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.241+0000 7fd051ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:44.233 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.241+0000 7fd051ffb640 10 client.?.objecter ms_handle_connect 0x7fd05c0b1f60 2024-03-29T10:56:44.233 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.241+0000 7fd051ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:44.233 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.241+0000 7fd051ffb640 1 -- 172.21.15.112:0/3391590031 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2578780319 0 0) 0x7fd05c14b050 con 0x7fd05c154750 2024-03-29T10:56:44.233 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.241+0000 7fd051ffb640 1 -- 172.21.15.112:0/3391590031 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd040004eb0 con 0x7fd05c154750 2024-03-29T10:56:44.233 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.241+0000 7fd051ffb640 1 -- 172.21.15.112:0/3391590031 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fd054001bd0 con 0x7fd05c154750 2024-03-29T10:56:44.233 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.241+0000 7fd051ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:44.234 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.241+0000 7fd051ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:44.234 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.241+0000 7fd051ffb640 10 monclient(hunting): dump: 2024-03-29T10:56:44.234 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:44.234 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:44.234 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:44.234 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:44.234 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:44.234 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:44.234 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:44.234 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:44.234 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:44.234 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:44.234 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd051ffb640 1 -- 172.21.15.112:0/3391590031 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2532976642 0 0) 0x7fd05c1402d0 con 0x7fd05c143d10 2024-03-29T10:56:44.235 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd051ffb640 1 -- 172.21.15.112:0/3391590031 >> v1:172.21.15.118:6789/0 conn(0x7fd05c0b1f60 legacy=0x7fd05c13f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:44.235 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd051ffb640 1 -- 172.21.15.112:0/3391590031 >> v1:172.21.15.112:6789/0 conn(0x7fd05c154750 legacy=0x7fd05c149b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:44.235 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd051ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:56:44.235 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd051ffb640 1 monclient: found mon.c 2024-03-29T10:56:44.235 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd051ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:44.235 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd051ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:44.235 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd051ffb640 1 -- 172.21.15.112:0/3391590031 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05c14b330 con 0x7fd05c143d10 2024-03-29T10:56:44.235 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd051ffb640 10 monclient: _finish_auth 0 2024-03-29T10:56:44.236 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd051ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:56:44.236 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd051ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:44.236 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd063304640 5 monclient: authenticate success, global_id 4436 2024-03-29T10:56:44.236 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd063304640 10 monclient: _renew_subs 2024-03-29T10:56:44.236 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd063304640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:44.236 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd063304640 1 -- 172.21.15.112:0/3391590031 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd05c147be0 con 0x7fd05c143d10 2024-03-29T10:56:44.236 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd063304640 10 client.4436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:44.236 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd063304640 10 monclient: _renew_subs 2024-03-29T10:56:44.237 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd063304640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:44.237 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd063304640 1 -- 172.21.15.112:0/3391590031 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd05c147790 con 0x7fd05c143d10 2024-03-29T10:56:44.237 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd063304640 1 librados: init done 2024-03-29T10:56:44.237 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd051ffb640 1 -- 172.21.15.112:0/3391590031 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd04c001d60 con 0x7fd05c143d10 2024-03-29T10:56:44.237 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd051ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:44.237 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd051ffb640 1 -- 172.21.15.112:0/3391590031 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fd04c0037b0 con 0x7fd05c143d10 2024-03-29T10:56:44.237 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd051ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:44.237 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd051ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:44.237 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd051ffb640 10 monclient: dump: 2024-03-29T10:56:44.237 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:44.237 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:44.237 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:44.238 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:44.238 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:44.238 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:44.238 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:44.238 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:44.238 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:44.238 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:44.238 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.242+0000 7fd052ffd640 4 set_mon_vals no callback set 2024-03-29T10:56:44.238 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.244+0000 7fd0477fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:44.238 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.244+0000 7fd0477fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:44.238 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.244+0000 7fd0477fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:44.238 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.244+0000 7fd0477fe640 1 -- 172.21.15.112:0/3391590031 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd05c147a40 con 0x7fd05c143d10 2024-03-29T10:56:44.238 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.244+0000 7fd051ffb640 1 -- 172.21.15.112:0/3391590031 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fd04c003910 con 0x7fd05c143d10 2024-03-29T10:56:44.238 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.245+0000 7fd051ffb640 1 -- 172.21.15.112:0/3391590031 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7fd04c083780 con 0x7fd05c143d10 2024-03-29T10:56:44.239 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.245+0000 7fd051ffb640 10 client.4436.objecter ms_dispatch 0x7fd05c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:44.239 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.245+0000 7fd051ffb640 3 client.4436.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:44.239 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.245+0000 7fd051ffb640 3 client.4436.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:44.239 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.245+0000 7fd051ffb640 20 client.4436.objecter dump_active .. 0 homeless 2024-03-29T10:56:44.239 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.245+0000 7fd0610a0640 10 monclient: get_auth_request con 0x7fd04006df30 auth_method 0 2024-03-29T10:56:44.239 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.246+0000 7fd051ffb640 10 client.4436.objecter ms_handle_connect 0x7fd04006df30 2024-03-29T10:56:44.239 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.252+0000 7fd051ffb640 1 -- 172.21.15.112:0/3391590031 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd05c147a40 con 0x7fd05c143d10 2024-03-29T10:56:44.239 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.252+0000 7fd051ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:44.239 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.252+0000 7fd051ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:44.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.629+0000 7fd0477fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:44.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.629+0000 7fd0477fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:44.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.629+0000 7fd0477fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:44.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.629+0000 7fd0477fe640 1 -- 172.21.15.112:0/3391590031 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd05c147a40 con 0x7fd05c143d10 2024-03-29T10:56:44.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.631+0000 7fd051ffb640 1 -- 172.21.15.112:0/3391590031 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7fd05c147a40 con 0x7fd05c143d10 2024-03-29T10:56:44.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.631+0000 7fd051ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:44.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.631+0000 7fd051ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:44.612 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:56:44.612 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:44.615 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.634+0000 7fd0477fe640 10 librados: watch_flush enter 2024-03-29T10:56:44.615 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.634+0000 7fd0477fe640 10 librados: watch_flush exit 2024-03-29T10:56:44.615 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.634+0000 7fd0477fe640 20 client.4436.objecter shutdown clearing up homeless session... 2024-03-29T10:56:44.615 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.634+0000 7fd0477fe640 10 client.4436.objecter successfully canceled tick 2024-03-29T10:56:44.616 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.635+0000 7fd0477fe640 1 -- 172.21.15.112:0/3391590031 >> v1:172.21.15.118:6816/3181073408 conn(0x7fd04006df30 legacy=0x7fd0400703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:44.616 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.635+0000 7fd0477fe640 10 monclient: shutdown 2024-03-29T10:56:44.616 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.635+0000 7fd0477fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:44.616 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.635+0000 7fd0477fe640 1 -- 172.21.15.112:0/3391590031 >> v1:172.21.15.112:6790/0 conn(0x7fd05c143d10 legacy=0x7fd05c149420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:44.616 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.635+0000 7fd0477fe640 1 -- 172.21.15.112:0/3391590031 shutdown_connections 2024-03-29T10:56:44.616 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.635+0000 7fd0477fe640 1 -- 172.21.15.112:0/3391590031 >> 172.21.15.112:0/3391590031 conn(0x7fd05c066a00 msgr2=0x7fd05c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:44.616 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.635+0000 7fd0477fe640 1 -- 172.21.15.112:0/3391590031 shutdown_connections 2024-03-29T10:56:44.616 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.635+0000 7fd0477fe640 1 -- 172.21.15.112:0/3391590031 wait complete. 2024-03-29T10:56:44.616 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:44.635+0000 7fd0477fe640 1 librados: shutdown 2024-03-29T10:56:44.631 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:51.437 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:51.586 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.606+0000 7f24cb59e640 10 monclient: get_monmap_and_config 2024-03-29T10:56:51.586 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.606+0000 7f24cb59e640 10 monclient: build_initial_monmap 2024-03-29T10:56:51.587 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.606+0000 7f24cb59e640 10 monclient: monmap: 2024-03-29T10:56:51.587 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:56:51.587 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:51.587 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:56:51.607016+0000 2024-03-29T10:56:51.587 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:56:51.607016+0000 2024-03-29T10:56:51.587 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:51.587 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:51.587 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:56:51.587 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:56:51.587 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:56:51.587 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:51.590 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.610+0000 7f24cb59e640 1 Processor -- start 2024-03-29T10:56:51.590 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.610+0000 7f24cb59e640 1 -- start start 2024-03-29T10:56:51.590 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.610+0000 7f24cb59e640 10 monclient: init 2024-03-29T10:56:51.592 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.611+0000 7f24cb59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:51.592 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.611+0000 7f24cb59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:56:51.592 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.611+0000 7f24cb59e640 10 monclient(hunting): picked mon.noname-a con 0x7f24c4006a70 addr v1:172.21.15.112:6789/0 2024-03-29T10:56:51.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.611+0000 7f24cb59e640 10 monclient(hunting): picked mon.noname-c con 0x7f24c4033060 addr v1:172.21.15.118:6789/0 2024-03-29T10:56:51.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.614+0000 7f24ca59c640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f24c4006a70 0x7f24c40c7a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:39848/0 (socket says 172.21.15.112:39848) 2024-03-29T10:56:51.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.615+0000 7f24ca59c640 1 -- 172.21.15.112:0/3578369756 learned_addr learned my addr 172.21.15.112:0/3578369756 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:56:51.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.615+0000 7f24cb59e640 10 monclient(hunting): picked mon.noname-b con 0x7f24c40c1e70 addr v1:172.21.15.112:6790/0 2024-03-29T10:56:51.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.615+0000 7f24cb59e640 1 -- 172.21.15.112:0/3578369756 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c402cfb0 con 0x7f24c4006a70 2024-03-29T10:56:51.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.615+0000 7f24cb59e640 1 -- 172.21.15.112:0/3578369756 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c402ae70 con 0x7f24c4033060 2024-03-29T10:56:51.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.615+0000 7f24cb59e640 1 -- 172.21.15.112:0/3578369756 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c40c8110 con 0x7f24c40c1e70 2024-03-29T10:56:51.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.615+0000 7f24cb59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:51.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.615+0000 7f24cb59e640 10 monclient(hunting): authenticate will time out at 1166.919902s 2024-03-29T10:56:51.598 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.617+0000 7f24c959a640 1 -- 172.21.15.112:0/3578369756 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254568585 0 0) 0x7f24c402cfb0 con 0x7f24c4006a70 2024-03-29T10:56:51.598 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.617+0000 7f24c959a640 10 monclient(hunting): _negotiate 2024-03-29T10:56:51.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.617+0000 7f24c959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:51.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.617+0000 7f24c959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:51.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.617+0000 7f24c959a640 10 monclient(hunting): my global_id is 4413 2024-03-29T10:56:51.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.617+0000 7f24c959a640 1 -- 172.21.15.112:0/3578369756 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24b4005730 con 0x7f24c4006a70 2024-03-29T10:56:51.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.618+0000 7f24c959a640 1 -- 172.21.15.112:0/3578369756 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3670307934 0 0) 0x7f24b4005730 con 0x7f24c4006a70 2024-03-29T10:56:51.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.618+0000 7f24c959a640 1 -- 172.21.15.112:0/3578369756 >> v1:172.21.15.112:6790/0 conn(0x7f24c40c1e70 legacy=0x7f24c40cfda0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:51.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.618+0000 7f24c959a640 1 -- 172.21.15.112:0/3578369756 >> v1:172.21.15.118:6789/0 conn(0x7f24c4033060 legacy=0x7f24c40c1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:51.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.618+0000 7f24c959a640 10 monclient: _finish_hunting 0 2024-03-29T10:56:51.600 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.618+0000 7f24c959a640 1 monclient: found mon.noname-a 2024-03-29T10:56:51.600 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.618+0000 7f24c959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:51.600 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.618+0000 7f24c959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T10:56:51.600 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.618+0000 7f24c959a640 1 -- 172.21.15.112:0/3578369756 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c402d200 con 0x7f24c4006a70 2024-03-29T10:56:51.600 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.618+0000 7f24c959a640 10 monclient: _finish_auth 0 2024-03-29T10:56:51.600 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.618+0000 7f24c959a640 10 monclient: _check_auth_tickets 2024-03-29T10:56:51.600 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.618+0000 7f24c959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:51.600 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.619+0000 7f24c959a640 1 -- 172.21.15.112:0/3578369756 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f24cc04f730 con 0x7f24c4006a70 2024-03-29T10:56:51.600 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.619+0000 7f24c959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:51.600 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.619+0000 7f24c959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:56:51.600 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.619+0000 7f24c959a640 10 monclient: dump: 2024-03-29T10:56:51.600 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:51.600 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:51.600 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:51.601 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:51.601 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:51.601 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:51.601 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:51.601 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:51.601 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:51.601 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:51.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.619+0000 7f24cb59e640 5 monclient: authenticate success, global_id 4413 2024-03-29T10:56:51.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.619+0000 7f24cb59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:51.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.619+0000 7f24c959a640 1 -- 172.21.15.112:0/3578369756 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24cc0008d0 con 0x7f24c4006a70 2024-03-29T10:56:51.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.619+0000 7f24c959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:51.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.619+0000 7f24c959a640 1 -- 172.21.15.112:0/3578369756 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f24cc0758b0 con 0x7f24c4006a70 2024-03-29T10:56:51.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.619+0000 7f24c959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:51.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.619+0000 7f24c959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:51.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.619+0000 7f24c959a640 10 monclient: dump: 2024-03-29T10:56:51.602 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:51.602 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:51.602 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:51.602 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:51.602 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:51.602 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:51.602 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:51.602 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:51.602 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:51.602 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:51.602 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.619+0000 7f24cb59e640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:51.602 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.619+0000 7f24cb59e640 4 set_mon_vals no callback set 2024-03-29T10:56:51.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.620+0000 7f24cb59e640 10 monclient: shutdown 2024-03-29T10:56:51.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.620+0000 7f24cb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:51.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.620+0000 7f24cb59e640 1 -- 172.21.15.112:0/3578369756 >> v1:172.21.15.112:6789/0 conn(0x7f24c4006a70 legacy=0x7f24c40c7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:51.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.621+0000 7f24cb59e640 1 -- 172.21.15.112:0/3578369756 shutdown_connections 2024-03-29T10:56:51.604 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.621+0000 7f24cb59e640 1 -- 172.21.15.112:0/3578369756 >> 172.21.15.112:0/3578369756 conn(0x7f24c40058f0 msgr2=0x7f24c4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:51.604 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.622+0000 7f24cb59e640 1 -- 172.21.15.112:0/3578369756 shutdown_connections 2024-03-29T10:56:51.604 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.622+0000 7f24cb59e640 1 -- 172.21.15.112:0/3578369756 wait complete. 2024-03-29T10:56:51.604 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.623+0000 7f24cb59e640 10 monclient: build_initial_monmap 2024-03-29T10:56:51.604 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.624+0000 7f24cb59e640 10 monclient: monmap: 2024-03-29T10:56:51.604 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:56:51.604 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:51.604 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:56:51.605 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:56:51.605 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:51.605 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:51.605 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:56:51.605 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:56:51.605 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:56:51.605 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:51.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.624+0000 7f24cb59e640 1 librados: starting msgr at 2024-03-29T10:56:51.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.624+0000 7f24cb59e640 1 librados: starting objecter 2024-03-29T10:56:51.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.624+0000 7f24cb59e640 1 Processor -- start 2024-03-29T10:56:51.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.625+0000 7f24cb59e640 1 -- start start 2024-03-29T10:56:51.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.625+0000 7f24cb59e640 1 librados: setting wanted keys 2024-03-29T10:56:51.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.625+0000 7f24cb59e640 1 librados: calling monclient init 2024-03-29T10:56:51.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.625+0000 7f24cb59e640 10 monclient: init 2024-03-29T10:56:51.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.626+0000 7f24cb59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:51.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.626+0000 7f24cb59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:56:51.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.626+0000 7f24cb59e640 10 monclient(hunting): picked mon.noname-a con 0x7f24c4006a70 addr v1:172.21.15.112:6789/0 2024-03-29T10:56:51.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.626+0000 7f24ca59c640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f24c4006a70 0x7f24c402de50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:40420/0 (socket says 172.21.15.112:40420) 2024-03-29T10:56:51.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.626+0000 7f24ca59c640 1 -- 172.21.15.112:0/2894976685 learned_addr learned my addr 172.21.15.112:0/2894976685 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:56:51.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.626+0000 7f24baffd640 10 client.?.objecter ms_handle_connect 0x7f24c4006a70 2024-03-29T10:56:51.607 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.626+0000 7f24baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:51.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.630+0000 7f24cb59e640 10 monclient(hunting): picked mon.noname-b con 0x7f24c40c1e70 addr v1:172.21.15.118:6789/0 2024-03-29T10:56:51.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.630+0000 7f24cb59e640 10 monclient(hunting): picked mon.noname-c con 0x7f24c40c4fc0 addr v1:172.21.15.112:6790/0 2024-03-29T10:56:51.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.630+0000 7f24cb59e640 1 -- 172.21.15.112:0/2894976685 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c402f4e0 con 0x7f24c4006a70 2024-03-29T10:56:51.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.630+0000 7f24cb59e640 1 -- 172.21.15.112:0/2894976685 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c402f750 con 0x7f24c40c1e70 2024-03-29T10:56:51.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.630+0000 7f24cb59e640 1 -- 172.21.15.112:0/2894976685 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c415b660 con 0x7f24c40c4fc0 2024-03-29T10:56:51.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.630+0000 7f24cb59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:51.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.630+0000 7f24cb59e640 10 monclient(hunting): authenticate will time out at 1166.934699s 2024-03-29T10:56:51.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.631+0000 7f24baffd640 1 -- 172.21.15.112:0/2894976685 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 189460131 0 0) 0x7f24c402f4e0 con 0x7f24c4006a70 2024-03-29T10:56:51.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.631+0000 7f24baffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:51.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.631+0000 7f24baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:51.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.631+0000 7f24baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:51.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.631+0000 7f24baffd640 10 monclient(hunting): my global_id is 4416 2024-03-29T10:56:51.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.631+0000 7f24baffd640 1 -- 172.21.15.112:0/2894976685 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24ac004eb0 con 0x7f24c4006a70 2024-03-29T10:56:51.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.631+0000 7f24baffd640 1 -- 172.21.15.112:0/2894976685 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3030773044 0 0) 0x7f24ac004eb0 con 0x7f24c4006a70 2024-03-29T10:56:51.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.631+0000 7f24baffd640 1 -- 172.21.15.112:0/2894976685 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24c402f4e0 con 0x7f24c4006a70 2024-03-29T10:56:51.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.631+0000 7f24baffd640 1 -- 172.21.15.112:0/2894976685 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f24cc04f400 con 0x7f24c4006a70 2024-03-29T10:56:51.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.631+0000 7f24baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:51.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.631+0000 7f24baffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:56:51.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.631+0000 7f24baffd640 10 monclient(hunting): dump: 2024-03-29T10:56:51.613 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:51.613 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:51.613 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:51.613 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:51.613 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:51.613 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:51.614 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:51.614 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:51.614 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:51.614 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:51.614 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.632+0000 7f24baffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:51.620 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.632+0000 7f24ca59c640 1 -- 172.21.15.112:0/2894976685 >> v1:172.21.15.112:6789/0 conn(0x7f24c4006a70 legacy=0x7f24c402de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T10:56:51.620 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.632+0000 7f24ca59c640 1 -- 172.21.15.112:0/2894976685 >> v1:172.21.15.112:6789/0 conn(0x7f24c4006a70 legacy=0x7f24c402de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:56:51.620 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.632+0000 7f24ca59c640 1 --1- 172.21.15.112:0/2894976685 >> v1:172.21.15.112:6789/0 conn(0x7f24c4006a70 0x7f24c402de50 :-1 s=OPENED pgs=568276 cs=1 l=1).handle_message read tag failed 2024-03-29T10:56:51.620 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.632+0000 7f24ca59c640 1 --1- 172.21.15.112:0/2894976685 >> v1:172.21.15.112:6789/0 conn(0x7f24c4006a70 0x7f24c402de50 :-1 s=OPENED pgs=568276 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:56:51.620 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.633+0000 7f24baffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.112:6789/0 2024-03-29T10:56:51.620 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.633+0000 7f24baffd640 10 client.?.objecter ms_handle_connect 0x7f24c40c1e70 2024-03-29T10:56:51.620 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.633+0000 7f24baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:51.620 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.633+0000 7f24baffd640 1 -- 172.21.15.112:0/2894976685 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950039994 0 0) 0x7f24c402f750 con 0x7f24c40c1e70 2024-03-29T10:56:51.620 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.633+0000 7f24baffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:51.620 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.633+0000 7f24baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:51.620 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.633+0000 7f24baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:51.620 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.633+0000 7f24baffd640 10 monclient(hunting): my global_id is 4405 2024-03-29T10:56:51.620 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.634+0000 7f24baffd640 1 -- 172.21.15.112:0/2894976685 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24ac004eb0 con 0x7f24c40c1e70 2024-03-29T10:56:51.621 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.634+0000 7f24baffd640 10 client.?.objecter ms_handle_connect 0x7f24c40c4fc0 2024-03-29T10:56:51.621 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.634+0000 7f24baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:51.621 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.634+0000 7f24baffd640 1 -- 172.21.15.112:0/2894976685 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2283324745 0 0) 0x7f24ac004eb0 con 0x7f24c40c1e70 2024-03-29T10:56:51.621 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.634+0000 7f24baffd640 1 -- 172.21.15.112:0/2894976685 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24c402f750 con 0x7f24c40c1e70 2024-03-29T10:56:51.621 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.634+0000 7f24baffd640 1 -- 172.21.15.112:0/2894976685 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060687086 0 0) 0x7f24c415b660 con 0x7f24c40c4fc0 2024-03-29T10:56:51.621 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.634+0000 7f24baffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:51.621 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.634+0000 7f24baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:51.621 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.635+0000 7f24baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:51.621 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.635+0000 7f24baffd640 10 monclient(hunting): my global_id is 4439 2024-03-29T10:56:51.621 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.635+0000 7f24baffd640 1 -- 172.21.15.112:0/2894976685 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24ac004eb0 con 0x7f24c40c4fc0 2024-03-29T10:56:51.621 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.635+0000 7f24baffd640 1 -- 172.21.15.112:0/2894976685 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f24c0001870 con 0x7f24c40c1e70 2024-03-29T10:56:51.621 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.635+0000 7f24baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:51.621 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.635+0000 7f24baffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:51.621 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.635+0000 7f24baffd640 10 monclient(hunting): dump: 2024-03-29T10:56:51.621 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:51.622 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:51.622 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:51.622 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:51.622 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:51.622 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:51.622 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:51.622 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:51.622 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:51.622 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:51.622 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.635+0000 7f24baffd640 1 -- 172.21.15.112:0/2894976685 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1714487001 0 0) 0x7f24c402f750 con 0x7f24c40c1e70 2024-03-29T10:56:51.622 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.635+0000 7f24baffd640 1 -- 172.21.15.112:0/2894976685 >> v1:172.21.15.112:6790/0 conn(0x7f24c40c4fc0 legacy=0x7f24c40bae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:51.622 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.635+0000 7f24baffd640 1 -- 172.21.15.112:0/2894976685 >> v1:172.21.15.112:6789/0 conn(0x7f24c4006a70 legacy=0x7f24c402de50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:56:51.622 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.635+0000 7f24baffd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:51.622 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.635+0000 7f24baffd640 1 monclient: found mon.b 2024-03-29T10:56:51.623 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.635+0000 7f24baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:51.623 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.636+0000 7f24baffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:56:51.623 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.636+0000 7f24baffd640 1 -- 172.21.15.112:0/2894976685 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c40c2400 con 0x7f24c40c1e70 2024-03-29T10:56:51.623 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.636+0000 7f24baffd640 10 monclient: _finish_auth 0 2024-03-29T10:56:51.623 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.636+0000 7f24baffd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:51.623 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.636+0000 7f24baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:51.623 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.636+0000 7f24baffd640 1 -- 172.21.15.112:0/2894976685 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24c0000fe0 con 0x7f24c40c1e70 2024-03-29T10:56:51.623 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.636+0000 7f24cb59e640 5 monclient: authenticate success, global_id 4405 2024-03-29T10:56:51.623 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.636+0000 7f24cb59e640 10 monclient: _renew_subs 2024-03-29T10:56:51.623 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.636+0000 7f24cb59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:56:51.623 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.636+0000 7f24cb59e640 1 -- 172.21.15.112:0/2894976685 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c415c290 con 0x7f24c40c1e70 2024-03-29T10:56:51.623 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.636+0000 7f24cb59e640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:51.623 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.636+0000 7f24baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:51.623 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.636+0000 7f24bbfff640 4 set_mon_vals no callback set 2024-03-29T10:56:51.623 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.637+0000 7f24baffd640 1 -- 172.21.15.112:0/2894976685 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f24c0003610 con 0x7f24c40c1e70 2024-03-29T10:56:51.624 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.637+0000 7f24baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:51.624 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.637+0000 7f24baffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:51.624 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.637+0000 7f24baffd640 10 monclient: dump: 2024-03-29T10:56:51.624 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:51.624 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:51.624 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:51.624 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:51.624 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:51.624 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:51.624 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:51.624 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:51.624 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:51.624 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:51.624 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.637+0000 7f24cb59e640 10 monclient: _renew_subs 2024-03-29T10:56:51.624 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.637+0000 7f24cb59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:56:51.625 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.637+0000 7f24cb59e640 1 -- 172.21.15.112:0/2894976685 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c40bb700 con 0x7f24c40c1e70 2024-03-29T10:56:51.625 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.637+0000 7f24cb59e640 1 librados: init done 2024-03-29T10:56:51.625 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.638+0000 7f24cb59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:51.625 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.638+0000 7f24cb59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:51.625 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.638+0000 7f24cb59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:56:51.625 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.638+0000 7f24cb59e640 1 -- 172.21.15.112:0/2894976685 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c415c480 con 0x7f24c40c1e70 2024-03-29T10:56:51.625 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.639+0000 7f24baffd640 1 -- 172.21.15.112:0/2894976685 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f24c00195e0 con 0x7f24c40c1e70 2024-03-29T10:56:51.626 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.640+0000 7f24baffd640 1 -- 172.21.15.112:0/2894976685 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f24c0083690 con 0x7f24c40c1e70 2024-03-29T10:56:51.626 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.640+0000 7f24baffd640 10 client.4405.objecter ms_dispatch 0x7f24cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:51.626 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.640+0000 7f24baffd640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:51.626 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.640+0000 7f24baffd640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:51.626 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.640+0000 7f24baffd640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-29T10:56:51.626 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.641+0000 7f24ca59c640 10 monclient: get_auth_request con 0x7f24ac06dc60 auth_method 0 2024-03-29T10:56:51.626 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.641+0000 7f24baffd640 10 client.4405.objecter ms_handle_connect 0x7f24ac06dc60 2024-03-29T10:56:51.628 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.647+0000 7f24baffd640 1 -- 172.21.15.112:0/2894976685 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f24c004eda0 con 0x7f24c40c1e70 2024-03-29T10:56:51.628 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.647+0000 7f24baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:51.628 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.647+0000 7f24baffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:51.961 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.978+0000 7f24cb59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:51.961 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.978+0000 7f24cb59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:51.961 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.978+0000 7f24cb59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:56:51.961 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.978+0000 7f24cb59e640 1 -- 172.21.15.112:0/2894976685 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24c4004cd0 con 0x7f24c40c1e70 2024-03-29T10:56:51.966 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.985+0000 7f24baffd640 1 -- 172.21.15.112:0/2894976685 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f24c0050ea0 con 0x7f24c40c1e70 2024-03-29T10:56:51.966 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.985+0000 7f24baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:51.966 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.985+0000 7f24baffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:51.967 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:56:51.967 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:51.970 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.989+0000 7f24cb59e640 10 librados: watch_flush enter 2024-03-29T10:56:51.970 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.989+0000 7f24cb59e640 10 librados: watch_flush exit 2024-03-29T10:56:51.970 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.989+0000 7f24cb59e640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-29T10:56:51.970 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.989+0000 7f24cb59e640 10 client.4405.objecter successfully canceled tick 2024-03-29T10:56:51.971 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.990+0000 7f24cb59e640 1 -- 172.21.15.112:0/2894976685 >> v1:172.21.15.118:6816/3181073408 conn(0x7f24ac06dc60 legacy=0x7f24ac070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:51.971 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.990+0000 7f24cb59e640 10 monclient: shutdown 2024-03-29T10:56:51.971 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.990+0000 7f24cb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:51.971 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.990+0000 7f24cb59e640 1 -- 172.21.15.112:0/2894976685 >> v1:172.21.15.118:6789/0 conn(0x7f24c40c1e70 legacy=0x7f24c415bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:51.972 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.992+0000 7f24cb59e640 1 -- 172.21.15.112:0/2894976685 shutdown_connections 2024-03-29T10:56:51.972 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.992+0000 7f24cb59e640 1 -- 172.21.15.112:0/2894976685 >> 172.21.15.112:0/2894976685 conn(0x7f24c40058f0 msgr2=0x7f24c40050f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:51.973 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.992+0000 7f24cb59e640 1 -- 172.21.15.112:0/2894976685 shutdown_connections 2024-03-29T10:56:51.973 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.992+0000 7f24cb59e640 1 -- 172.21.15.112:0/2894976685 wait complete. 2024-03-29T10:56:51.973 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:51.993+0000 7f24cb59e640 1 librados: shutdown 2024-03-29T10:56:51.990 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:58.798 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:58.948 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.964+0000 7f080683e640 10 monclient: get_monmap_and_config 2024-03-29T10:56:58.948 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.964+0000 7f080683e640 10 monclient: build_initial_monmap 2024-03-29T10:56:58.948 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.964+0000 7f080683e640 10 monclient: monmap: 2024-03-29T10:56:58.948 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:56:58.948 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:58.948 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:56:58.965013+0000 2024-03-29T10:56:58.948 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:56:58.965013+0000 2024-03-29T10:56:58.948 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:58.948 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:58.948 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:56:58.948 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:56:58.948 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:56:58.949 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:58.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.965+0000 7f080683e640 1 Processor -- start 2024-03-29T10:56:58.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.965+0000 7f080683e640 1 -- start start 2024-03-29T10:56:58.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.965+0000 7f080683e640 10 monclient: init 2024-03-29T10:56:58.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.966+0000 7f080683e640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:58.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.966+0000 7f080683e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:56:58.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.966+0000 7f080683e640 10 monclient(hunting): picked mon.noname-b con 0x7f08000ab360 addr v1:172.21.15.112:6790/0 2024-03-29T10:56:58.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.966+0000 7f080683e640 10 monclient(hunting): picked mon.noname-a con 0x7f080014fb00 addr v1:172.21.15.112:6789/0 2024-03-29T10:56:58.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.966+0000 7f07fffff640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f08000ab360 0x7f08000ad450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:48662/0 (socket says 172.21.15.112:48662) 2024-03-29T10:56:58.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.966+0000 7f080683e640 10 monclient(hunting): picked mon.noname-c con 0x7f0800153090 addr v1:172.21.15.118:6789/0 2024-03-29T10:56:58.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.966+0000 7f080683e640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08000a94e0 con 0x7f080014fb00 2024-03-29T10:56:58.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.966+0000 7f080683e640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08000afb60 con 0x7f0800153090 2024-03-29T10:56:58.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.966+0000 7f080683e640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08000aab80 con 0x7f08000ab360 2024-03-29T10:56:58.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.966+0000 7f080683e640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:58.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.966+0000 7f080683e640 10 monclient(hunting): authenticate will time out at 1174.270860s 2024-03-29T10:56:58.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.966+0000 7f07fffff640 1 -- 172.21.15.112:0/2431394279 learned_addr learned my addr 172.21.15.112:0/2431394279 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:56:58.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.968+0000 7f07feffd640 1 -- 172.21.15.112:0/2431394279 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248635806 0 0) 0x7f08000aab80 con 0x7f08000ab360 2024-03-29T10:56:58.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.968+0000 7f07feffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:58.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.968+0000 7f07feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:58.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.968+0000 7f07feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:58.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.968+0000 7f07feffd640 10 monclient(hunting): my global_id is 4442 2024-03-29T10:56:58.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.968+0000 7f07feffd640 1 -- 172.21.15.112:0/2431394279 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07e8003520 con 0x7f08000ab360 2024-03-29T10:56:58.953 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 1 -- 172.21.15.112:0/2431394279 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1140555141 0 0) 0x7f08000a94e0 con 0x7f080014fb00 2024-03-29T10:56:58.953 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:58.953 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:58.953 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:58.953 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 10 monclient(hunting): my global_id is 4419 2024-03-29T10:56:58.953 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 1 -- 172.21.15.112:0/2431394279 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08000aab80 con 0x7f080014fb00 2024-03-29T10:56:58.953 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 1 -- 172.21.15.112:0/2431394279 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2671622178 0 0) 0x7f08000afb60 con 0x7f0800153090 2024-03-29T10:56:58.953 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:58.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:58.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:58.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 10 monclient(hunting): my global_id is 4408 2024-03-29T10:56:58.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 1 -- 172.21.15.112:0/2431394279 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08000a94e0 con 0x7f0800153090 2024-03-29T10:56:58.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 1 -- 172.21.15.112:0/2431394279 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 704931414 0 0) 0x7f07e8003520 con 0x7f08000ab360 2024-03-29T10:56:58.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 1 -- 172.21.15.112:0/2431394279 >> v1:172.21.15.118:6789/0 conn(0x7f0800153090 legacy=0x7f080015cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:58.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 1 -- 172.21.15.112:0/2431394279 >> v1:172.21.15.112:6789/0 conn(0x7f080014fb00 legacy=0x7f0800152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:58.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:58.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 1 monclient: found mon.noname-b 2024-03-29T10:56:58.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:58.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:56:58.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 1 -- 172.21.15.112:0/2431394279 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08001568d0 con 0x7f08000ab360 2024-03-29T10:56:58.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 10 monclient: _finish_auth 0 2024-03-29T10:56:58.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:58.954 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:58.955 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f080683e640 5 monclient: authenticate success, global_id 4442 2024-03-29T10:56:58.955 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f080683e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:58.955 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 1 -- 172.21.15.112:0/2431394279 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f07f4001610 con 0x7f08000ab360 2024-03-29T10:56:58.955 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:58.955 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:58.955 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f07feffd640 10 monclient: dump: 2024-03-29T10:56:58.955 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:58.955 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:58.955 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:58.955 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:58.955 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:58.955 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:58.955 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:58.956 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:58.956 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:58.956 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:58.956 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.972+0000 7f080683e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:58.959 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.973+0000 7f07feffd640 1 -- 172.21.15.112:0/2431394279 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07f4001830 con 0x7f08000ab360 2024-03-29T10:56:58.959 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.973+0000 7f07feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:58.959 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.973+0000 7f080683e640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:58.959 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.973+0000 7f080683e640 4 set_mon_vals no callback set 2024-03-29T10:56:58.959 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.973+0000 7f07feffd640 1 -- 172.21.15.112:0/2431394279 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f07f4001bc0 con 0x7f08000ab360 2024-03-29T10:56:58.959 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.973+0000 7f07feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:58.959 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.973+0000 7f07feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:58.959 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.973+0000 7f07feffd640 10 monclient: dump: 2024-03-29T10:56:58.959 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:58.959 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:58.960 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:58.960 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:58.960 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:58.960 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:58.960 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:58.960 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:58.960 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:58.960 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:58.960 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.974+0000 7f080683e640 10 monclient: shutdown 2024-03-29T10:56:58.960 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.974+0000 7f080683e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:58.960 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.974+0000 7f080683e640 1 -- 172.21.15.112:0/2431394279 >> v1:172.21.15.112:6790/0 conn(0x7f08000ab360 legacy=0x7f08000ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:58.960 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.974+0000 7f080683e640 1 -- 172.21.15.112:0/2431394279 shutdown_connections 2024-03-29T10:56:58.960 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.974+0000 7f080683e640 1 -- 172.21.15.112:0/2431394279 >> 172.21.15.112:0/2431394279 conn(0x7f08000b35e0 msgr2=0x7f08000b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:58.960 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.974+0000 7f080683e640 1 -- 172.21.15.112:0/2431394279 shutdown_connections 2024-03-29T10:56:58.961 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.974+0000 7f080683e640 1 -- 172.21.15.112:0/2431394279 wait complete. 2024-03-29T10:56:58.961 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.975+0000 7f080683e640 10 monclient: build_initial_monmap 2024-03-29T10:56:58.961 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.975+0000 7f080683e640 10 monclient: monmap: 2024-03-29T10:56:58.961 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:56:58.961 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:58.961 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:56:58.961 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:56:58.961 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:58.961 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:58.961 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:56:58.961 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:56:58.961 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:56:58.961 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:58.961 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.975+0000 7f080683e640 1 librados: starting msgr at 2024-03-29T10:56:58.962 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.975+0000 7f080683e640 1 librados: starting objecter 2024-03-29T10:56:58.962 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.975+0000 7f080683e640 1 Processor -- start 2024-03-29T10:56:58.962 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.975+0000 7f080683e640 1 -- start start 2024-03-29T10:56:58.962 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.975+0000 7f080683e640 1 librados: setting wanted keys 2024-03-29T10:56:58.962 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.975+0000 7f080683e640 1 librados: calling monclient init 2024-03-29T10:56:58.962 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.975+0000 7f080683e640 10 monclient: init 2024-03-29T10:56:58.962 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.976+0000 7f080683e640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:58.962 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.976+0000 7f080683e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:56:58.962 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.976+0000 7f080683e640 10 monclient(hunting): picked mon.noname-b con 0x7f08000ab360 addr v1:172.21.15.118:6789/0 2024-03-29T10:56:58.962 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.976+0000 7f080683e640 10 monclient(hunting): picked mon.noname-a con 0x7f080014fb00 addr v1:172.21.15.112:6789/0 2024-03-29T10:56:58.962 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.976+0000 7f080683e640 10 monclient(hunting): picked mon.noname-c con 0x7f0800153090 addr v1:172.21.15.112:6790/0 2024-03-29T10:56:58.962 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.976+0000 7f080683e640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f080008ae90 con 0x7f080014fb00 2024-03-29T10:56:58.962 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.976+0000 7f080683e640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08000701d0 con 0x7f08000ab360 2024-03-29T10:56:58.963 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.976+0000 7f080683e640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08000704b0 con 0x7f0800153090 2024-03-29T10:56:58.963 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.976+0000 7f080683e640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:58.963 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.976+0000 7f080683e640 10 monclient(hunting): authenticate will time out at 1174.280974s 2024-03-29T10:56:58.963 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.977+0000 7f0804ddb640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f0800153090 0x7f08001f3bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:49776/0 (socket says 172.21.15.112:49776) 2024-03-29T10:56:58.963 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.977+0000 7f0804ddb640 1 -- 172.21.15.112:0/2792028417 learned_addr learned my addr 172.21.15.112:0/2792028417 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:56:58.970 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.979+0000 7f07fcff9640 10 client.?.objecter ms_handle_connect 0x7f0800153090 2024-03-29T10:56:58.970 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.979+0000 7f07fcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:58.970 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.979+0000 7f07fcff9640 1 -- 172.21.15.112:0/2792028417 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160677591 0 0) 0x7f08000704b0 con 0x7f0800153090 2024-03-29T10:56:58.970 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.979+0000 7f07fcff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:58.970 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.979+0000 7f07fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:58.970 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.979+0000 7f07fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:58.971 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.979+0000 7f07fcff9640 10 monclient(hunting): my global_id is 4445 2024-03-29T10:56:58.971 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.979+0000 7f07fcff9640 1 -- 172.21.15.112:0/2792028417 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07ec004eb0 con 0x7f0800153090 2024-03-29T10:56:58.971 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.980+0000 7f07fcff9640 10 client.?.objecter ms_handle_connect 0x7f08000ab360 2024-03-29T10:56:58.971 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.980+0000 7f07fcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:58.971 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.980+0000 7f07fcff9640 1 -- 172.21.15.112:0/2792028417 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557823821 0 0) 0x7f08000701d0 con 0x7f08000ab360 2024-03-29T10:56:58.971 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.980+0000 7f07fcff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:58.971 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.980+0000 7f07fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:58.971 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.980+0000 7f07fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:58.971 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.980+0000 7f07fcff9640 10 monclient(hunting): my global_id is 4411 2024-03-29T10:56:58.971 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.980+0000 7f07fcff9640 1 -- 172.21.15.112:0/2792028417 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08000704b0 con 0x7f08000ab360 2024-03-29T10:56:58.971 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.981+0000 7f07fcff9640 10 client.?.objecter ms_handle_connect 0x7f080014fb00 2024-03-29T10:56:58.971 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.981+0000 7f07fcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:58.971 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.981+0000 7f07fcff9640 1 -- 172.21.15.112:0/2792028417 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3535489579 0 0) 0x7f08000704b0 con 0x7f08000ab360 2024-03-29T10:56:58.972 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.981+0000 7f07fcff9640 1 -- 172.21.15.112:0/2792028417 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08000701d0 con 0x7f08000ab360 2024-03-29T10:56:58.972 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.981+0000 7f07fcff9640 1 -- 172.21.15.112:0/2792028417 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f07f4001550 con 0x7f08000ab360 2024-03-29T10:56:58.972 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.981+0000 7f07fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:58.972 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.981+0000 7f07fcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:58.972 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.981+0000 7f07fcff9640 10 monclient(hunting): dump: 2024-03-29T10:56:58.972 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:58.972 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:58.972 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:58.972 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:58.972 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:58.972 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:58.972 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:58.972 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:58.972 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:58.973 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:58.973 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.981+0000 7f07fcff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:58.973 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.982+0000 7f07fcff9640 1 -- 172.21.15.112:0/2792028417 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3301342624 0 0) 0x7f07ec004eb0 con 0x7f0800153090 2024-03-29T10:56:58.973 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.982+0000 7f07fcff9640 1 -- 172.21.15.112:0/2792028417 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08000704b0 con 0x7f0800153090 2024-03-29T10:56:58.973 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.982+0000 7f07fcff9640 1 -- 172.21.15.112:0/2792028417 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339905714 0 0) 0x7f080008ae90 con 0x7f080014fb00 2024-03-29T10:56:58.973 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.982+0000 7f07fcff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:58.973 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.982+0000 7f07fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:58.973 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.982+0000 7f07fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:58.973 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.982+0000 7f07fcff9640 10 monclient(hunting): my global_id is 4422 2024-03-29T10:56:58.973 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.982+0000 7f07fcff9640 1 -- 172.21.15.112:0/2792028417 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07ec004eb0 con 0x7f080014fb00 2024-03-29T10:56:58.973 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.982+0000 7f07fcff9640 1 -- 172.21.15.112:0/2792028417 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f07f8001b90 con 0x7f0800153090 2024-03-29T10:56:58.973 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.982+0000 7f07fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:58.973 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.982+0000 7f07fcff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:58.973 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.982+0000 7f07fcff9640 10 monclient(hunting): dump: 2024-03-29T10:56:58.974 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:58.974 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:58.974 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:58.974 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:58.974 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:58.974 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:58.974 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:58.974 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:58.974 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:58.974 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:58.974 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.982+0000 7f07fcff9640 1 -- 172.21.15.112:0/2792028417 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2471625456 0 0) 0x7f08000704b0 con 0x7f0800153090 2024-03-29T10:56:58.974 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.982+0000 7f07fcff9640 1 -- 172.21.15.112:0/2792028417 >> v1:172.21.15.118:6789/0 conn(0x7f08000ab360 legacy=0x7f080006e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:58.974 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.982+0000 7f07fcff9640 1 -- 172.21.15.112:0/2792028417 >> v1:172.21.15.112:6789/0 conn(0x7f080014fb00 legacy=0x7f08001f04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:58.974 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.982+0000 7f07fcff9640 10 monclient: _finish_hunting 0 2024-03-29T10:56:58.975 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.982+0000 7f07fcff9640 1 monclient: found mon.c 2024-03-29T10:56:58.975 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.982+0000 7f07fcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:58.975 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.982+0000 7f07fcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:58.975 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.982+0000 7f07fcff9640 1 -- 172.21.15.112:0/2792028417 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080008b0e0 con 0x7f0800153090 2024-03-29T10:56:58.975 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.982+0000 7f07fcff9640 10 monclient: _finish_auth 0 2024-03-29T10:56:58.975 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.982+0000 7f07fcff9640 10 monclient: _check_auth_tickets 2024-03-29T10:56:58.975 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.982+0000 7f07fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:58.975 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.983+0000 7f080683e640 5 monclient: authenticate success, global_id 4445 2024-03-29T10:56:58.975 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.983+0000 7f080683e640 10 monclient: _renew_subs 2024-03-29T10:56:58.975 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.983+0000 7f080683e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:58.975 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.983+0000 7f080683e640 1 -- 172.21.15.112:0/2792028417 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f080006eeb0 con 0x7f0800153090 2024-03-29T10:56:58.975 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.983+0000 7f080683e640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:58.975 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.983+0000 7f080683e640 10 monclient: _renew_subs 2024-03-29T10:56:58.975 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.983+0000 7f080683e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:58.976 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.983+0000 7f080683e640 1 -- 172.21.15.112:0/2792028417 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08001566a0 con 0x7f0800153090 2024-03-29T10:56:58.976 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.983+0000 7f080683e640 1 librados: init done 2024-03-29T10:56:58.976 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.983+0000 7f07fcff9640 1 -- 172.21.15.112:0/2792028417 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07f8001cf0 con 0x7f0800153090 2024-03-29T10:56:58.976 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.983+0000 7f07fcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:58.976 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.983+0000 7f07fcff9640 1 -- 172.21.15.112:0/2792028417 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f07f8003b30 con 0x7f0800153090 2024-03-29T10:56:58.976 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.983+0000 7f07fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:58.976 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.983+0000 7f07fcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:58.976 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.983+0000 7f07fe7fc640 4 set_mon_vals no callback set 2024-03-29T10:56:58.976 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.983+0000 7f07fcff9640 10 monclient: dump: 2024-03-29T10:56:58.976 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:56:58.976 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:56:58.976 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:58.976 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:56:58.976 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:58.977 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:56:58.977 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:56:58.977 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:56:58.977 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:56:58.977 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:56:58.977 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.986+0000 7f080683e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:58.977 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.986+0000 7f080683e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:58.977 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.986+0000 7f080683e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:58.977 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.986+0000 7f080683e640 1 -- 172.21.15.112:0/2792028417 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07e0005080 con 0x7f0800153090 2024-03-29T10:56:58.977 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.987+0000 7f07fcff9640 1 -- 172.21.15.112:0/2792028417 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f07f8003c90 con 0x7f0800153090 2024-03-29T10:56:58.977 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.988+0000 7f07fffff640 10 monclient: get_auth_request con 0x7f07ec06df30 auth_method 0 2024-03-29T10:56:58.977 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.988+0000 7f07fcff9640 1 -- 172.21.15.112:0/2792028417 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f07f8084c80 con 0x7f0800153090 2024-03-29T10:56:58.977 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.988+0000 7f07fcff9640 10 client.4445.objecter ms_dispatch 0x7f0800000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:58.977 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.988+0000 7f07fcff9640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:58.978 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.988+0000 7f07fcff9640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:58.978 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.988+0000 7f07fcff9640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-29T10:56:58.978 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.988+0000 7f07fcff9640 10 client.4445.objecter ms_handle_connect 0x7f07ec06df30 2024-03-29T10:56:58.978 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.996+0000 7f07fcff9640 1 -- 172.21.15.112:0/2792028417 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f07f8050560 con 0x7f0800153090 2024-03-29T10:56:58.978 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.996+0000 7f07fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:58.979 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:58.996+0000 7f07fcff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:59.301 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:59.320+0000 7f080683e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:59.301 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:59.320+0000 7f080683e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:59.301 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:59.320+0000 7f080683e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:56:59.301 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:59.320+0000 7f080683e640 1 -- 172.21.15.112:0/2792028417 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07e0005330 con 0x7f0800153090 2024-03-29T10:56:59.303 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:59.322+0000 7f07fcff9640 1 -- 172.21.15.112:0/2792028417 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f07f8052490 con 0x7f0800153090 2024-03-29T10:56:59.303 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:59.322+0000 7f07fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:59.303 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:59.322+0000 7f07fcff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:59.303 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:56:59.303 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:59.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:59.326+0000 7f080683e640 10 librados: watch_flush enter 2024-03-29T10:56:59.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:59.326+0000 7f080683e640 10 librados: watch_flush exit 2024-03-29T10:56:59.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:59.326+0000 7f080683e640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-29T10:56:59.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:59.326+0000 7f080683e640 10 client.4445.objecter successfully canceled tick 2024-03-29T10:56:59.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:59.326+0000 7f080683e640 1 -- 172.21.15.112:0/2792028417 >> v1:172.21.15.118:6816/3181073408 conn(0x7f07ec06df30 legacy=0x7f07ec0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:59.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:59.326+0000 7f080683e640 10 monclient: shutdown 2024-03-29T10:56:59.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:59.326+0000 7f080683e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:59.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:59.326+0000 7f080683e640 1 -- 172.21.15.112:0/2792028417 >> v1:172.21.15.112:6790/0 conn(0x7f0800153090 legacy=0x7f08001f3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:59.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:59.327+0000 7f080683e640 1 -- 172.21.15.112:0/2792028417 shutdown_connections 2024-03-29T10:56:59.308 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:59.327+0000 7f080683e640 1 -- 172.21.15.112:0/2792028417 >> 172.21.15.112:0/2792028417 conn(0x7f08000b35e0 msgr2=0x7f08000b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:59.308 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:59.327+0000 7f080683e640 1 -- 172.21.15.112:0/2792028417 shutdown_connections 2024-03-29T10:56:59.308 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:59.327+0000 7f080683e640 1 -- 172.21.15.112:0/2792028417 wait complete. 2024-03-29T10:56:59.308 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:56:59.327+0000 7f080683e640 1 librados: shutdown 2024-03-29T10:56:59.325 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:06.130 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:06.283 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.296+0000 7f22d4b96640 10 monclient: get_monmap_and_config 2024-03-29T10:57:06.283 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.296+0000 7f22d4b96640 10 monclient: build_initial_monmap 2024-03-29T10:57:06.283 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.296+0000 7f22d4b96640 10 monclient: monmap: 2024-03-29T10:57:06.283 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:57:06.283 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:06.284 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:57:06.297187+0000 2024-03-29T10:57:06.284 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:57:06.297187+0000 2024-03-29T10:57:06.284 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:06.284 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:06.284 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:57:06.284 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:57:06.284 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:57:06.284 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:06.284 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.297+0000 7f22d4b96640 1 Processor -- start 2024-03-29T10:57:06.284 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.297+0000 7f22d4b96640 1 -- start start 2024-03-29T10:57:06.284 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.297+0000 7f22d4b96640 10 monclient: init 2024-03-29T10:57:06.284 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.298+0000 7f22d4b96640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:06.284 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.298+0000 7f22d4b96640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:06.284 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.298+0000 7f22d4b96640 10 monclient(hunting): picked mon.noname-b con 0x7f22d00ab360 addr v1:172.21.15.112:6790/0 2024-03-29T10:57:06.285 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.298+0000 7f22d4b96640 10 monclient(hunting): picked mon.noname-a con 0x7f22d014fb00 addr v1:172.21.15.112:6789/0 2024-03-29T10:57:06.285 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.298+0000 7f22d4b96640 10 monclient(hunting): picked mon.noname-c con 0x7f22d0153090 addr v1:172.21.15.118:6789/0 2024-03-29T10:57:06.285 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.298+0000 7f22d4b96640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22d00a94e0 con 0x7f22d014fb00 2024-03-29T10:57:06.285 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.298+0000 7f22d4b96640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22d00afb60 con 0x7f22d0153090 2024-03-29T10:57:06.285 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.298+0000 7f22d4b96640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22d00aab80 con 0x7f22d00ab360 2024-03-29T10:57:06.285 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.298+0000 7f22d4b96640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:06.285 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.298+0000 7f22d4b96640 10 monclient(hunting): authenticate will time out at 1181.602701s 2024-03-29T10:57:06.285 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.302+0000 7f22cdd9b640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f22d014fb00 0x7f22d0152980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:57721/0 (socket says 172.21.15.112:57721) 2024-03-29T10:57:06.285 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.302+0000 7f22cdd9b640 1 -- 172.21.15.112:0/1521080957 learned_addr learned my addr 172.21.15.112:0/1521080957 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:57:06.286 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.304+0000 7f22cd59a640 1 -- 172.21.15.112:0/1521080957 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3351111634 0 0) 0x7f22d00a94e0 con 0x7f22d014fb00 2024-03-29T10:57:06.286 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:57:06.286 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:06.286 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:06.287 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 10 monclient(hunting): my global_id is 4425 2024-03-29T10:57:06.287 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 1 -- 172.21.15.112:0/1521080957 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22b8003520 con 0x7f22d014fb00 2024-03-29T10:57:06.287 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 1 -- 172.21.15.112:0/1521080957 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290808272 0 0) 0x7f22d00aab80 con 0x7f22d00ab360 2024-03-29T10:57:06.287 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:57:06.287 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:06.287 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:06.287 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 10 monclient(hunting): my global_id is 4448 2024-03-29T10:57:06.287 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 1 -- 172.21.15.112:0/1521080957 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22d00a94e0 con 0x7f22d00ab360 2024-03-29T10:57:06.287 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 1 -- 172.21.15.112:0/1521080957 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183608048 0 0) 0x7f22d00afb60 con 0x7f22d0153090 2024-03-29T10:57:06.287 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:57:06.287 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:06.287 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:06.287 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 10 monclient(hunting): my global_id is 4414 2024-03-29T10:57:06.287 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 1 -- 172.21.15.112:0/1521080957 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22d00aab80 con 0x7f22d0153090 2024-03-29T10:57:06.288 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 1 -- 172.21.15.112:0/1521080957 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1282781456 0 0) 0x7f22d00a94e0 con 0x7f22d00ab360 2024-03-29T10:57:06.288 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 1 -- 172.21.15.112:0/1521080957 >> v1:172.21.15.118:6789/0 conn(0x7f22d0153090 legacy=0x7f22d015cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:06.288 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 1 -- 172.21.15.112:0/1521080957 >> v1:172.21.15.112:6789/0 conn(0x7f22d014fb00 legacy=0x7f22d0152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:06.288 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 10 monclient: _finish_hunting 0 2024-03-29T10:57:06.288 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 1 monclient: found mon.noname-b 2024-03-29T10:57:06.288 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:06.288 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:57:06.288 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 1 -- 172.21.15.112:0/1521080957 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d01568d0 con 0x7f22d00ab360 2024-03-29T10:57:06.288 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 10 monclient: _finish_auth 0 2024-03-29T10:57:06.288 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 10 monclient: _check_auth_tickets 2024-03-29T10:57:06.288 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:06.288 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22d4b96640 5 monclient: authenticate success, global_id 4448 2024-03-29T10:57:06.288 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22d4b96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:06.288 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 1 -- 172.21.15.112:0/1521080957 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f22c80019f0 con 0x7f22d00ab360 2024-03-29T10:57:06.289 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:06.289 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:06.289 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 10 monclient: dump: 2024-03-29T10:57:06.289 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:06.289 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:06.289 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:06.289 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:06.289 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:06.289 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:06.289 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:06.289 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:06.289 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:06.289 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:06.289 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22d4b96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:06.290 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 1 -- 172.21.15.112:0/1521080957 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22c8001c40 con 0x7f22d00ab360 2024-03-29T10:57:06.290 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:06.290 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22d4b96640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:06.290 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22d4b96640 4 set_mon_vals no callback set 2024-03-29T10:57:06.290 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.305+0000 7f22cd59a640 1 -- 172.21.15.112:0/1521080957 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f22c8001fd0 con 0x7f22d00ab360 2024-03-29T10:57:06.290 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.306+0000 7f22cd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:06.290 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.306+0000 7f22cd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:06.290 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.306+0000 7f22cd59a640 10 monclient: dump: 2024-03-29T10:57:06.290 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:06.290 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:06.290 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:06.290 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:06.290 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:06.290 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:06.291 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:06.291 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:06.291 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:06.291 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:06.292 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.306+0000 7f22d4b96640 10 monclient: shutdown 2024-03-29T10:57:06.292 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.306+0000 7f22d4b96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:06.292 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.306+0000 7f22d4b96640 1 -- 172.21.15.112:0/1521080957 >> v1:172.21.15.112:6790/0 conn(0x7f22d00ab360 legacy=0x7f22d00ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:06.292 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.309+0000 7f22d4b96640 1 -- 172.21.15.112:0/1521080957 shutdown_connections 2024-03-29T10:57:06.292 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.309+0000 7f22d4b96640 1 -- 172.21.15.112:0/1521080957 >> 172.21.15.112:0/1521080957 conn(0x7f22d00b35e0 msgr2=0x7f22d00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:06.292 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.309+0000 7f22d4b96640 1 -- 172.21.15.112:0/1521080957 shutdown_connections 2024-03-29T10:57:06.292 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.310+0000 7f22d4b96640 1 -- 172.21.15.112:0/1521080957 wait complete. 2024-03-29T10:57:06.296 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.311+0000 7f22d4b96640 10 monclient: build_initial_monmap 2024-03-29T10:57:06.296 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.311+0000 7f22d4b96640 10 monclient: monmap: 2024-03-29T10:57:06.296 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:57:06.296 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:06.296 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:57:06.296 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:57:06.296 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:06.296 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:06.297 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:57:06.297 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:57:06.297 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:57:06.297 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:06.298 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.311+0000 7f22d4b96640 1 librados: starting msgr at 2024-03-29T10:57:06.298 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.311+0000 7f22d4b96640 1 librados: starting objecter 2024-03-29T10:57:06.298 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.311+0000 7f22d4b96640 1 Processor -- start 2024-03-29T10:57:06.298 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.311+0000 7f22d4b96640 1 -- start start 2024-03-29T10:57:06.298 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.311+0000 7f22d4b96640 1 librados: setting wanted keys 2024-03-29T10:57:06.298 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.311+0000 7f22d4b96640 1 librados: calling monclient init 2024-03-29T10:57:06.298 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.311+0000 7f22d4b96640 10 monclient: init 2024-03-29T10:57:06.298 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.312+0000 7f22d4b96640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:06.298 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.312+0000 7f22d4b96640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:06.298 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.312+0000 7f22d4b96640 10 monclient(hunting): picked mon.noname-c con 0x7f22d00ab360 addr v1:172.21.15.112:6790/0 2024-03-29T10:57:06.298 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.312+0000 7f22d4b96640 10 monclient(hunting): picked mon.noname-a con 0x7f22d014fb00 addr v1:172.21.15.112:6789/0 2024-03-29T10:57:06.298 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.312+0000 7f22d4b96640 10 monclient(hunting): picked mon.noname-b con 0x7f22d0153090 addr v1:172.21.15.118:6789/0 2024-03-29T10:57:06.298 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.312+0000 7f22d4b96640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22d00880d0 con 0x7f22d014fb00 2024-03-29T10:57:06.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.312+0000 7f22d4b96640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22d0088bb0 con 0x7f22d0153090 2024-03-29T10:57:06.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.312+0000 7f22d4b96640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22d0088e90 con 0x7f22d00ab360 2024-03-29T10:57:06.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.312+0000 7f22d4b96640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:06.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.312+0000 7f22d4b96640 10 monclient(hunting): authenticate will time out at 1181.616622s 2024-03-29T10:57:06.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.315+0000 7f22ce59c640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f22d00ab360 0x7f22d0156000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:48089/0 (socket says 172.21.15.112:48089) 2024-03-29T10:57:06.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.315+0000 7f22ce59c640 1 -- 172.21.15.112:0/1032314614 learned_addr learned my addr 172.21.15.112:0/1032314614 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:57:06.301 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.318+0000 7f22beffd640 10 client.?.objecter ms_handle_connect 0x7f22d00ab360 2024-03-29T10:57:06.301 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.318+0000 7f22beffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:06.301 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.319+0000 7f22beffd640 10 client.?.objecter ms_handle_connect 0x7f22d0153090 2024-03-29T10:57:06.302 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.319+0000 7f22beffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:06.302 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.319+0000 7f22beffd640 1 -- 172.21.15.112:0/1032314614 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617632128 0 0) 0x7f22d0088e90 con 0x7f22d00ab360 2024-03-29T10:57:06.302 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.319+0000 7f22beffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:06.302 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.319+0000 7f22beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:06.302 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.319+0000 7f22beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:06.302 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.319+0000 7f22beffd640 10 monclient(hunting): my global_id is 4451 2024-03-29T10:57:06.302 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.319+0000 7f22beffd640 1 -- 172.21.15.112:0/1032314614 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22b0004eb0 con 0x7f22d00ab360 2024-03-29T10:57:06.302 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.319+0000 7f22beffd640 10 client.?.objecter ms_handle_connect 0x7f22d014fb00 2024-03-29T10:57:06.302 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.319+0000 7f22beffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:06.302 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.319+0000 7f22beffd640 1 -- 172.21.15.112:0/1032314614 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643018669 0 0) 0x7f22d0088bb0 con 0x7f22d0153090 2024-03-29T10:57:06.302 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.319+0000 7f22beffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:06.302 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.319+0000 7f22beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:06.302 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.319+0000 7f22beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:06.302 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.319+0000 7f22beffd640 10 monclient(hunting): my global_id is 4417 2024-03-29T10:57:06.303 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.319+0000 7f22beffd640 1 -- 172.21.15.112:0/1032314614 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22d0088e90 con 0x7f22d0153090 2024-03-29T10:57:06.303 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.319+0000 7f22beffd640 1 -- 172.21.15.112:0/1032314614 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3141083878 0 0) 0x7f22b0004eb0 con 0x7f22d00ab360 2024-03-29T10:57:06.303 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.319+0000 7f22beffd640 1 -- 172.21.15.112:0/1032314614 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22d0088bb0 con 0x7f22d00ab360 2024-03-29T10:57:06.303 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.320+0000 7f22beffd640 1 -- 172.21.15.112:0/1032314614 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f22c8001f90 con 0x7f22d00ab360 2024-03-29T10:57:06.303 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.320+0000 7f22beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:06.303 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.320+0000 7f22beffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:06.303 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.320+0000 7f22beffd640 10 monclient(hunting): dump: 2024-03-29T10:57:06.303 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:06.303 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:06.303 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:06.303 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:06.303 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:06.303 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:06.303 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:06.304 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:06.304 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:06.304 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:06.305 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.320+0000 7f22beffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:06.305 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.321+0000 7f22beffd640 1 -- 172.21.15.112:0/1032314614 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906182624 0 0) 0x7f22d00880d0 con 0x7f22d014fb00 2024-03-29T10:57:06.305 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.321+0000 7f22beffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:06.305 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.321+0000 7f22beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:06.305 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.321+0000 7f22beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:06.306 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.321+0000 7f22beffd640 10 monclient(hunting): my global_id is 4428 2024-03-29T10:57:06.306 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.321+0000 7f22beffd640 1 -- 172.21.15.112:0/1032314614 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22b0004eb0 con 0x7f22d014fb00 2024-03-29T10:57:06.306 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.321+0000 7f22beffd640 1 -- 172.21.15.112:0/1032314614 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2035941299 0 0) 0x7f22d0088bb0 con 0x7f22d00ab360 2024-03-29T10:57:06.306 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.321+0000 7f22beffd640 1 -- 172.21.15.112:0/1032314614 >> v1:172.21.15.118:6789/0 conn(0x7f22d0153090 legacy=0x7f22d01ff8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:06.306 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.321+0000 7f22beffd640 1 -- 172.21.15.112:0/1032314614 >> v1:172.21.15.112:6789/0 conn(0x7f22d014fb00 legacy=0x7f22d0153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:06.306 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.321+0000 7f22beffd640 10 monclient: _finish_hunting 0 2024-03-29T10:57:06.306 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.321+0000 7f22beffd640 1 monclient: found mon.c 2024-03-29T10:57:06.306 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.321+0000 7f22beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:06.306 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.321+0000 7f22beffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:06.306 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.321+0000 7f22beffd640 1 -- 172.21.15.112:0/1032314614 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d0089170 con 0x7f22d00ab360 2024-03-29T10:57:06.306 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.321+0000 7f22beffd640 10 monclient: _finish_auth 0 2024-03-29T10:57:06.306 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.321+0000 7f22beffd640 10 monclient: _check_auth_tickets 2024-03-29T10:57:06.306 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.321+0000 7f22beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:06.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.322+0000 7f22d4b96640 5 monclient: authenticate success, global_id 4451 2024-03-29T10:57:06.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.322+0000 7f22d4b96640 10 monclient: _renew_subs 2024-03-29T10:57:06.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.322+0000 7f22d4b96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:06.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.322+0000 7f22d4b96640 1 -- 172.21.15.112:0/1032314614 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22d0087660 con 0x7f22d00ab360 2024-03-29T10:57:06.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.322+0000 7f22d4b96640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:06.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.322+0000 7f22d4b96640 10 monclient: _renew_subs 2024-03-29T10:57:06.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.322+0000 7f22d4b96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:06.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.322+0000 7f22d4b96640 1 -- 172.21.15.112:0/1032314614 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22d0153f80 con 0x7f22d00ab360 2024-03-29T10:57:06.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.322+0000 7f22d4b96640 1 librados: init done 2024-03-29T10:57:06.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.322+0000 7f22beffd640 1 -- 172.21.15.112:0/1032314614 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22c8003ce0 con 0x7f22d00ab360 2024-03-29T10:57:06.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.322+0000 7f22beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:06.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.323+0000 7f22beffd640 1 -- 172.21.15.112:0/1032314614 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f22c8004070 con 0x7f22d00ab360 2024-03-29T10:57:06.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.323+0000 7f22beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:06.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.323+0000 7f22beffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:06.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.323+0000 7f22beffd640 10 monclient: dump: 2024-03-29T10:57:06.308 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:06.308 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:06.308 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:06.308 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:06.308 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:06.308 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:06.308 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:06.308 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:06.308 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:06.308 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:06.308 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.323+0000 7f22ccd99640 4 set_mon_vals no callback set 2024-03-29T10:57:06.310 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.325+0000 7f22beffd640 1 -- 172.21.15.112:0/1032314614 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f22c80041d0 con 0x7f22d00ab360 2024-03-29T10:57:06.310 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.325+0000 7f22beffd640 1 -- 172.21.15.112:0/1032314614 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f22c80a6ed0 con 0x7f22d00ab360 2024-03-29T10:57:06.310 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.325+0000 7f22beffd640 10 client.4451.objecter ms_dispatch 0x7f22d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:06.310 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.325+0000 7f22beffd640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:06.310 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.325+0000 7f22beffd640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:06.310 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.325+0000 7f22beffd640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-29T10:57:06.310 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.326+0000 7f22cdd9b640 10 monclient: get_auth_request con 0x7f22b006ddc0 auth_method 0 2024-03-29T10:57:06.310 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.326+0000 7f22d4b96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:06.310 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.326+0000 7f22d4b96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:06.310 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.326+0000 7f22d4b96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:06.310 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.326+0000 7f22d4b96640 1 -- 172.21.15.112:0/1032314614 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22b4005080 con 0x7f22d00ab360 2024-03-29T10:57:06.310 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.326+0000 7f22beffd640 10 client.4451.objecter ms_handle_connect 0x7f22b006ddc0 2024-03-29T10:57:06.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.335+0000 7f22beffd640 1 -- 172.21.15.112:0/1032314614 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f22c80727b0 con 0x7f22d00ab360 2024-03-29T10:57:06.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.335+0000 7f22beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:06.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.335+0000 7f22beffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:06.644 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.663+0000 7f22d4b96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:06.644 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.663+0000 7f22d4b96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:06.645 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.663+0000 7f22d4b96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:06.645 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.663+0000 7f22d4b96640 1 -- 172.21.15.112:0/1032314614 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22b4005330 con 0x7f22d00ab360 2024-03-29T10:57:06.645 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.664+0000 7f22beffd640 1 -- 172.21.15.112:0/1032314614 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f22c80746e0 con 0x7f22d00ab360 2024-03-29T10:57:06.645 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.664+0000 7f22beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:06.646 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.664+0000 7f22beffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:06.646 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:57:06.646 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:06.648 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.668+0000 7f22d4b96640 10 librados: watch_flush enter 2024-03-29T10:57:06.649 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.669+0000 7f22d4b96640 10 librados: watch_flush exit 2024-03-29T10:57:06.649 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.669+0000 7f22d4b96640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-29T10:57:06.649 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.669+0000 7f22d4b96640 10 client.4451.objecter successfully canceled tick 2024-03-29T10:57:06.649 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.669+0000 7f22d4b96640 1 -- 172.21.15.112:0/1032314614 >> v1:172.21.15.118:6816/3181073408 conn(0x7f22b006ddc0 legacy=0x7f22b0070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:06.649 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.669+0000 7f22d4b96640 10 monclient: shutdown 2024-03-29T10:57:06.650 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.669+0000 7f22d4b96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:06.650 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.669+0000 7f22d4b96640 1 -- 172.21.15.112:0/1032314614 >> v1:172.21.15.112:6790/0 conn(0x7f22d00ab360 legacy=0x7f22d0156000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:06.651 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.671+0000 7f22d4b96640 1 -- 172.21.15.112:0/1032314614 shutdown_connections 2024-03-29T10:57:06.651 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.671+0000 7f22d4b96640 1 -- 172.21.15.112:0/1032314614 >> 172.21.15.112:0/1032314614 conn(0x7f22d00b35e0 msgr2=0x7f22d00aeaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:06.651 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.671+0000 7f22d4b96640 1 -- 172.21.15.112:0/1032314614 shutdown_connections 2024-03-29T10:57:06.652 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.671+0000 7f22d4b96640 1 -- 172.21.15.112:0/1032314614 wait complete. 2024-03-29T10:57:06.653 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:06.673+0000 7f22d4b96640 1 librados: shutdown 2024-03-29T10:57:06.671 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:13.477 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:13.634 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.653+0000 7f0b7f36d640 10 monclient: get_monmap_and_config 2024-03-29T10:57:13.634 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.653+0000 7f0b7f36d640 10 monclient: build_initial_monmap 2024-03-29T10:57:13.634 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.653+0000 7f0b7f36d640 10 monclient: monmap: 2024-03-29T10:57:13.635 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:57:13.635 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:13.635 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:57:13.654646+0000 2024-03-29T10:57:13.635 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:57:13.654646+0000 2024-03-29T10:57:13.635 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:13.635 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:13.635 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:57:13.635 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:57:13.635 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:57:13.635 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:13.636 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.655+0000 7f0b7f36d640 1 Processor -- start 2024-03-29T10:57:13.636 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.655+0000 7f0b7f36d640 1 -- start start 2024-03-29T10:57:13.636 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.655+0000 7f0b7f36d640 10 monclient: init 2024-03-29T10:57:13.636 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.656+0000 7f0b7f36d640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:13.636 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.656+0000 7f0b7f36d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:13.636 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.656+0000 7f0b7f36d640 10 monclient(hunting): picked mon.noname-c con 0x7f0b780a9fe0 addr v1:172.21.15.118:6789/0 2024-03-29T10:57:13.636 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.656+0000 7f0b7f36d640 10 monclient(hunting): picked mon.noname-b con 0x7f0b780af600 addr v1:172.21.15.112:6790/0 2024-03-29T10:57:13.637 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.656+0000 7f0b7f36d640 10 monclient(hunting): picked mon.noname-a con 0x7f0b780ad340 addr v1:172.21.15.112:6789/0 2024-03-29T10:57:13.637 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.656+0000 7f0b7f36d640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b780aaec0 con 0x7f0b780ad340 2024-03-29T10:57:13.637 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.656+0000 7f0b7f36d640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b780ab6a0 con 0x7f0b780a9fe0 2024-03-29T10:57:13.637 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.656+0000 7f0b7f36d640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b780b00a0 con 0x7f0b780af600 2024-03-29T10:57:13.637 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.656+0000 7f0b7f36d640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:13.637 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.656+0000 7f0b7f36d640 10 monclient(hunting): authenticate will time out at 1188.960634s 2024-03-29T10:57:13.640 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.659+0000 7f0b7e36b640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f0b780a9fe0 0x7f0b7813b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:50973/0 (socket says 172.21.15.112:50973) 2024-03-29T10:57:13.640 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.659+0000 7f0b7e36b640 1 -- 172.21.15.112:0/473171698 learned_addr learned my addr 172.21.15.112:0/473171698 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:57:13.643 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.661+0000 7f0b7d369640 1 -- 172.21.15.112:0/473171698 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2883945631 0 0) 0x7f0b780ab6a0 con 0x7f0b780a9fe0 2024-03-29T10:57:13.643 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.661+0000 7f0b7d369640 10 monclient(hunting): _negotiate 2024-03-29T10:57:13.643 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.661+0000 7f0b7d369640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:13.644 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.661+0000 7f0b7d369640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:13.644 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.661+0000 7f0b7d369640 10 monclient(hunting): my global_id is 4420 2024-03-29T10:57:13.644 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.661+0000 7f0b7d369640 1 -- 172.21.15.112:0/473171698 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b64003520 con 0x7f0b780a9fe0 2024-03-29T10:57:13.644 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.661+0000 7f0b7d369640 1 -- 172.21.15.112:0/473171698 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1252562563 0 0) 0x7f0b64003520 con 0x7f0b780a9fe0 2024-03-29T10:57:13.644 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.661+0000 7f0b7d369640 1 -- 172.21.15.112:0/473171698 >> v1:172.21.15.112:6790/0 conn(0x7f0b780af600 legacy=0x7f0b780acc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:57:13.644 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.661+0000 7f0b7d369640 1 -- 172.21.15.112:0/473171698 >> v1:172.21.15.112:6789/0 conn(0x7f0b780ad340 legacy=0x7f0b78149c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:57:13.644 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.661+0000 7f0b7d369640 10 monclient: _finish_hunting 0 2024-03-29T10:57:13.644 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.661+0000 7f0b7d369640 1 monclient: found mon.noname-c 2024-03-29T10:57:13.644 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.661+0000 7f0b7d369640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:13.644 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.661+0000 7f0b7d369640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T10:57:13.644 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.661+0000 7f0b7d369640 1 -- 172.21.15.112:0/473171698 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7813beb0 con 0x7f0b780a9fe0 2024-03-29T10:57:13.644 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.661+0000 7f0b7d369640 10 monclient: _finish_auth 0 2024-03-29T10:57:13.644 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.661+0000 7f0b7d369640 10 monclient: _check_auth_tickets 2024-03-29T10:57:13.644 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.661+0000 7f0b7d369640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:13.644 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.661+0000 7f0b7f36d640 5 monclient: authenticate success, global_id 4420 2024-03-29T10:57:13.645 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.661+0000 7f0b7f36d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:13.645 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.661+0000 7f0b7d369640 1 -- 172.21.15.112:0/473171698 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f0b74001690 con 0x7f0b780a9fe0 2024-03-29T10:57:13.645 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.661+0000 7f0b7d369640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:13.645 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.661+0000 7f0b7d369640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:13.645 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.661+0000 7f0b7d369640 10 monclient: dump: 2024-03-29T10:57:13.645 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:13.645 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:13.645 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:13.645 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:13.645 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:13.645 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:13.645 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:13.645 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:13.645 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:13.645 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:13.646 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.661+0000 7f0b7f36d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:13.646 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.662+0000 7f0b7d369640 1 -- 172.21.15.112:0/473171698 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b74001860 con 0x7f0b780a9fe0 2024-03-29T10:57:13.646 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.662+0000 7f0b7d369640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:13.646 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.662+0000 7f0b7f36d640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:13.646 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.662+0000 7f0b7f36d640 4 set_mon_vals no callback set 2024-03-29T10:57:13.646 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.662+0000 7f0b7d369640 1 -- 172.21.15.112:0/473171698 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f0b74001cb0 con 0x7f0b780a9fe0 2024-03-29T10:57:13.646 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.662+0000 7f0b7d369640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:13.646 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.662+0000 7f0b7d369640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:13.647 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.662+0000 7f0b7d369640 10 monclient: dump: 2024-03-29T10:57:13.647 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:13.647 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:13.647 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:13.647 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:13.647 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:13.647 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:13.647 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:13.647 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:13.647 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:13.647 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:13.647 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.663+0000 7f0b7f36d640 10 monclient: shutdown 2024-03-29T10:57:13.647 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.663+0000 7f0b7f36d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:13.647 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.663+0000 7f0b7f36d640 1 -- 172.21.15.112:0/473171698 >> v1:172.21.15.118:6789/0 conn(0x7f0b780a9fe0 legacy=0x7f0b7813b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:13.648 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.663+0000 7f0b7f36d640 1 -- 172.21.15.112:0/473171698 shutdown_connections 2024-03-29T10:57:13.648 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.663+0000 7f0b7f36d640 1 -- 172.21.15.112:0/473171698 >> 172.21.15.112:0/473171698 conn(0x7f0b780673e0 msgr2=0x7f0b780b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:13.648 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.663+0000 7f0b7f36d640 1 -- 172.21.15.112:0/473171698 shutdown_connections 2024-03-29T10:57:13.648 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.663+0000 7f0b7f36d640 1 -- 172.21.15.112:0/473171698 wait complete. 2024-03-29T10:57:13.648 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.664+0000 7f0b7f36d640 10 monclient: build_initial_monmap 2024-03-29T10:57:13.648 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.664+0000 7f0b7f36d640 10 monclient: monmap: 2024-03-29T10:57:13.648 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:57:13.648 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:13.648 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:57:13.648 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:57:13.648 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:13.648 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:13.648 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:57:13.648 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:57:13.648 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:57:13.649 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:13.649 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.665+0000 7f0b7f36d640 1 librados: starting msgr at 2024-03-29T10:57:13.649 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.665+0000 7f0b7f36d640 1 librados: starting objecter 2024-03-29T10:57:13.649 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.665+0000 7f0b7f36d640 1 Processor -- start 2024-03-29T10:57:13.649 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.667+0000 7f0b7f36d640 1 -- start start 2024-03-29T10:57:13.649 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.667+0000 7f0b7f36d640 1 librados: setting wanted keys 2024-03-29T10:57:13.649 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.667+0000 7f0b7f36d640 1 librados: calling monclient init 2024-03-29T10:57:13.650 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.667+0000 7f0b7f36d640 10 monclient: init 2024-03-29T10:57:13.650 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.668+0000 7f0b7f36d640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:13.650 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.668+0000 7f0b7f36d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:57:13.650 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.668+0000 7f0b7f36d640 10 monclient(hunting): picked mon.noname-b con 0x7f0b780a9fe0 addr v1:172.21.15.118:6789/0 2024-03-29T10:57:13.650 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.668+0000 7f0b7f36d640 10 monclient(hunting): picked mon.noname-a con 0x7f0b780ad340 addr v1:172.21.15.112:6789/0 2024-03-29T10:57:13.650 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.668+0000 7f0b7f36d640 10 monclient(hunting): picked mon.noname-c con 0x7f0b780af600 addr v1:172.21.15.112:6790/0 2024-03-29T10:57:13.650 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.668+0000 7f0b7f36d640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b781362b0 con 0x7f0b780ad340 2024-03-29T10:57:13.650 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.668+0000 7f0b7f36d640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b78135e60 con 0x7f0b780a9fe0 2024-03-29T10:57:13.650 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.668+0000 7f0b7f36d640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b78135ff0 con 0x7f0b780af600 2024-03-29T10:57:13.650 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.668+0000 7f0b7f36d640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:13.650 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.668+0000 7f0b7f36d640 10 monclient(hunting): authenticate will time out at 1188.972280s 2024-03-29T10:57:13.654 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.674+0000 7f0b7eb6c640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f0b780af600 0x7f0b78136ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:38537/0 (socket says 172.21.15.112:38537) 2024-03-29T10:57:13.654 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.674+0000 7f0b7eb6c640 1 -- 172.21.15.112:0/3838045119 learned_addr learned my addr 172.21.15.112:0/3838045119 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:57:13.654 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.674+0000 7f0b6effd640 10 client.?.objecter ms_handle_connect 0x7f0b780af600 2024-03-29T10:57:13.654 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.674+0000 7f0b6effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:13.655 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.674+0000 7f0b6effd640 1 -- 172.21.15.112:0/3838045119 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675553514 0 0) 0x7f0b78135ff0 con 0x7f0b780af600 2024-03-29T10:57:13.655 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.674+0000 7f0b6effd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:13.655 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.674+0000 7f0b6effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:13.655 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.674+0000 7f0b6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:13.655 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.674+0000 7f0b6effd640 10 monclient(hunting): my global_id is 4454 2024-03-29T10:57:13.655 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.674+0000 7f0b6effd640 1 -- 172.21.15.112:0/3838045119 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b5c004eb0 con 0x7f0b780af600 2024-03-29T10:57:13.656 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.675+0000 7f0b6effd640 1 -- 172.21.15.112:0/3838045119 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 727734750 0 0) 0x7f0b5c004eb0 con 0x7f0b780af600 2024-03-29T10:57:13.656 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.675+0000 7f0b6effd640 1 -- 172.21.15.112:0/3838045119 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b78135ff0 con 0x7f0b780af600 2024-03-29T10:57:13.656 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.675+0000 7f0b6effd640 1 -- 172.21.15.112:0/3838045119 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f0b70001c60 con 0x7f0b780af600 2024-03-29T10:57:13.656 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.675+0000 7f0b6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:13.656 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.675+0000 7f0b6effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:13.656 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.675+0000 7f0b6effd640 10 monclient(hunting): dump: 2024-03-29T10:57:13.656 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:13.656 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:13.656 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:13.656 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:13.657 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:13.657 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:13.657 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:13.657 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:13.657 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:13.657 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:13.657 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.675+0000 7f0b6effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:13.657 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.675+0000 7f0b6effd640 1 -- 172.21.15.112:0/3838045119 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3139669557 0 0) 0x7f0b78135ff0 con 0x7f0b780af600 2024-03-29T10:57:13.657 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.675+0000 7f0b6effd640 1 -- 172.21.15.112:0/3838045119 >> v1:172.21.15.118:6789/0 conn(0x7f0b780a9fe0 legacy=0x7f0b781d3bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:57:13.657 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.675+0000 7f0b6effd640 1 -- 172.21.15.112:0/3838045119 >> v1:172.21.15.112:6789/0 conn(0x7f0b780ad340 legacy=0x7f0b781367d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:57:13.659 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.676+0000 7f0b6effd640 10 monclient: _finish_hunting 0 2024-03-29T10:57:13.659 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.676+0000 7f0b6effd640 1 monclient: found mon.c 2024-03-29T10:57:13.659 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.676+0000 7f0b6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:13.659 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.676+0000 7f0b6effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:13.659 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.676+0000 7f0b6effd640 1 -- 172.21.15.112:0/3838045119 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b781e7bf0 con 0x7f0b780af600 2024-03-29T10:57:13.659 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.676+0000 7f0b6effd640 10 monclient: _finish_auth 0 2024-03-29T10:57:13.659 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.676+0000 7f0b6effd640 10 monclient: _check_auth_tickets 2024-03-29T10:57:13.660 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.676+0000 7f0b6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:13.660 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.677+0000 7f0b7f36d640 5 monclient: authenticate success, global_id 4454 2024-03-29T10:57:13.660 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.677+0000 7f0b7f36d640 10 monclient: _renew_subs 2024-03-29T10:57:13.660 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.677+0000 7f0b7f36d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:13.660 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.677+0000 7f0b7f36d640 1 -- 172.21.15.112:0/3838045119 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b781d4890 con 0x7f0b780af600 2024-03-29T10:57:13.660 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.677+0000 7f0b7f36d640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:13.660 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.677+0000 7f0b7f36d640 10 monclient: _renew_subs 2024-03-29T10:57:13.660 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.677+0000 7f0b7f36d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:13.660 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.677+0000 7f0b7f36d640 1 -- 172.21.15.112:0/3838045119 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b781e7320 con 0x7f0b780af600 2024-03-29T10:57:13.660 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.677+0000 7f0b7f36d640 1 librados: init done 2024-03-29T10:57:13.662 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.679+0000 7f0b6effd640 1 -- 172.21.15.112:0/3838045119 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b70003970 con 0x7f0b780af600 2024-03-29T10:57:13.662 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.679+0000 7f0b6effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:13.662 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.679+0000 7f0b7f36d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:13.662 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.679+0000 7f0b7f36d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:13.662 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.679+0000 7f0b7f36d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:13.662 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.679+0000 7f0b7f36d640 1 -- 172.21.15.112:0/3838045119 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b781364f0 con 0x7f0b780af600 2024-03-29T10:57:13.662 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.680+0000 7f0b6effd640 1 -- 172.21.15.112:0/3838045119 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f0b70001c60 con 0x7f0b780af600 2024-03-29T10:57:13.662 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.680+0000 7f0b6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:13.662 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.680+0000 7f0b6effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:13.662 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.680+0000 7f0b6effd640 10 monclient: dump: 2024-03-29T10:57:13.662 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:13.662 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:13.663 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:13.663 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:13.663 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:13.663 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:13.663 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:13.663 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:13.663 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:13.663 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:13.663 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.680+0000 7f0b7cb68640 4 set_mon_vals no callback set 2024-03-29T10:57:13.663 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.681+0000 7f0b6effd640 1 -- 172.21.15.112:0/3838045119 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f0b70003c60 con 0x7f0b780af600 2024-03-29T10:57:13.664 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.682+0000 7f0b6effd640 1 -- 172.21.15.112:0/3838045119 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f0b70054830 con 0x7f0b780af600 2024-03-29T10:57:13.664 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.682+0000 7f0b6effd640 10 client.4454.objecter ms_dispatch 0x7f0b78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:13.664 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.682+0000 7f0b6effd640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:13.664 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.682+0000 7f0b6effd640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:13.664 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.682+0000 7f0b6effd640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-29T10:57:13.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.684+0000 7f0b7e36b640 10 monclient: get_auth_request con 0x7f0b5c06dfa0 auth_method 0 2024-03-29T10:57:13.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.685+0000 7f0b6effd640 10 client.4454.objecter ms_handle_connect 0x7f0b5c06dfa0 2024-03-29T10:57:13.671 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.690+0000 7f0b6effd640 1 -- 172.21.15.112:0/3838045119 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b70050ad0 con 0x7f0b780af600 2024-03-29T10:57:13.671 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.690+0000 7f0b6effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:13.671 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:13.690+0000 7f0b6effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:13.999 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:14.019+0000 7f0b7f36d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:13.999 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:14.019+0000 7f0b7f36d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:13.999 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:14.019+0000 7f0b7f36d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:14.000 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:14.019+0000 7f0b7f36d640 1 -- 172.21.15.112:0/3838045119 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b781d4b90 con 0x7f0b780af600 2024-03-29T10:57:14.000 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:14.020+0000 7f0b6effd640 1 -- 172.21.15.112:0/3838045119 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f0b70050ad0 con 0x7f0b780af600 2024-03-29T10:57:14.000 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:14.020+0000 7f0b6effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:14.000 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:14.020+0000 7f0b6effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:14.001 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:57:14.001 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:14.004 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:14.024+0000 7f0b7f36d640 10 librados: watch_flush enter 2024-03-29T10:57:14.004 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:14.024+0000 7f0b7f36d640 10 librados: watch_flush exit 2024-03-29T10:57:14.004 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:14.024+0000 7f0b7f36d640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-29T10:57:14.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:14.024+0000 7f0b7f36d640 10 client.4454.objecter successfully canceled tick 2024-03-29T10:57:14.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:14.024+0000 7f0b7f36d640 1 -- 172.21.15.112:0/3838045119 >> v1:172.21.15.118:6816/3181073408 conn(0x7f0b5c06dfa0 legacy=0x7f0b5c070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:14.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:14.024+0000 7f0b7f36d640 10 monclient: shutdown 2024-03-29T10:57:14.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:14.024+0000 7f0b7f36d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:14.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:14.024+0000 7f0b7f36d640 1 -- 172.21.15.112:0/3838045119 >> v1:172.21.15.112:6790/0 conn(0x7f0b780af600 legacy=0x7f0b78136ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:14.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:14.025+0000 7f0b7f36d640 1 -- 172.21.15.112:0/3838045119 shutdown_connections 2024-03-29T10:57:14.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:14.025+0000 7f0b7f36d640 1 -- 172.21.15.112:0/3838045119 >> 172.21.15.112:0/3838045119 conn(0x7f0b780673e0 msgr2=0x7f0b78069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:14.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:14.025+0000 7f0b7f36d640 1 -- 172.21.15.112:0/3838045119 shutdown_connections 2024-03-29T10:57:14.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:14.025+0000 7f0b7f36d640 1 -- 172.21.15.112:0/3838045119 wait complete. 2024-03-29T10:57:14.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:14.025+0000 7f0b7f36d640 1 librados: shutdown 2024-03-29T10:57:14.025 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:20.832 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:20.979 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:20.997+0000 7f4140650640 10 monclient: get_monmap_and_config 2024-03-29T10:57:20.979 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:20.997+0000 7f4140650640 10 monclient: build_initial_monmap 2024-03-29T10:57:20.979 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:20.997+0000 7f4140650640 10 monclient: monmap: 2024-03-29T10:57:20.979 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:57:20.979 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:20.979 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:57:20.999045+0000 2024-03-29T10:57:20.979 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:57:20.999045+0000 2024-03-29T10:57:20.979 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:20.979 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:20.979 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:57:20.980 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:57:20.980 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:57:20.980 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:20.980 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:20.998+0000 7f4140650640 1 Processor -- start 2024-03-29T10:57:20.980 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:20.998+0000 7f4140650640 1 -- start start 2024-03-29T10:57:20.980 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:20.998+0000 7f4140650640 10 monclient: init 2024-03-29T10:57:20.982 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:20.999+0000 7f4140650640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:20.982 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:20.999+0000 7f4140650640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:57:20.982 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:20.999+0000 7f4140650640 10 monclient(hunting): picked mon.noname-a con 0x7f41380683f0 addr v1:172.21.15.112:6789/0 2024-03-29T10:57:20.982 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:20.999+0000 7f4140650640 10 monclient(hunting): picked mon.noname-c con 0x7f413813e960 addr v1:172.21.15.118:6789/0 2024-03-29T10:57:20.982 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:20.999+0000 7f4140650640 10 monclient(hunting): picked mon.noname-b con 0x7f413813b030 addr v1:172.21.15.112:6790/0 2024-03-29T10:57:20.983 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:20.999+0000 7f4140650640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413813e4e0 con 0x7f41380683f0 2024-03-29T10:57:20.983 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:20.999+0000 7f4140650640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41380702f0 con 0x7f413813e960 2024-03-29T10:57:20.983 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:20.999+0000 7f4140650640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41380705d0 con 0x7f413813b030 2024-03-29T10:57:20.983 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:20.999+0000 7f4140650640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:20.983 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:20.999+0000 7f4140650640 10 monclient(hunting): authenticate will time out at 1196.305046s 2024-03-29T10:57:20.983 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.002+0000 7f413ebed640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f413813b030 0x7f413813b480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:42522/0 (socket says 172.21.15.112:42522) 2024-03-29T10:57:20.983 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.002+0000 7f413ebed640 1 -- 172.21.15.112:0/3737570664 learned_addr learned my addr 172.21.15.112:0/3737570664 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:57:20.986 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.004+0000 7f413d3ea640 1 -- 172.21.15.112:0/3737570664 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901148295 0 0) 0x7f4128001110 con 0x7f41380683f0 2024-03-29T10:57:20.986 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.004+0000 7f413d3ea640 10 monclient(hunting): _negotiate 2024-03-29T10:57:20.986 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.004+0000 7f413d3ea640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:20.986 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.004+0000 7f413d3ea640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:20.986 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.004+0000 7f413d3ea640 10 monclient(hunting): my global_id is 4431 2024-03-29T10:57:20.986 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.004+0000 7f413d3ea640 1 -- 172.21.15.112:0/3737570664 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4124003520 con 0x7f41380683f0 2024-03-29T10:57:20.986 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.005+0000 7f413d3ea640 1 -- 172.21.15.112:0/3737570664 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270349250 0 0) 0x7f41380702f0 con 0x7f413813e960 2024-03-29T10:57:20.986 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.005+0000 7f413d3ea640 10 monclient(hunting): _negotiate 2024-03-29T10:57:20.987 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.005+0000 7f413d3ea640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:20.987 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.005+0000 7f413d3ea640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:20.987 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.005+0000 7f413d3ea640 10 monclient(hunting): my global_id is 4423 2024-03-29T10:57:20.987 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.005+0000 7f413d3ea640 1 -- 172.21.15.112:0/3737570664 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4128001110 con 0x7f413813e960 2024-03-29T10:57:20.987 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.005+0000 7f413d3ea640 1 -- 172.21.15.112:0/3737570664 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 23962579 0 0) 0x7f4128001110 con 0x7f413813e960 2024-03-29T10:57:20.987 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.005+0000 7f413d3ea640 1 -- 172.21.15.112:0/3737570664 >> v1:172.21.15.112:6790/0 conn(0x7f413813b030 legacy=0x7f413813b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:20.987 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.005+0000 7f413d3ea640 1 -- 172.21.15.112:0/3737570664 >> v1:172.21.15.112:6789/0 conn(0x7f41380683f0 legacy=0x7f41380ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:20.987 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.005+0000 7f413d3ea640 10 monclient: _finish_hunting 0 2024-03-29T10:57:20.987 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.005+0000 7f413d3ea640 1 monclient: found mon.noname-c 2024-03-29T10:57:20.988 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.005+0000 7f413d3ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:20.988 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.005+0000 7f413d3ea640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T10:57:20.988 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.005+0000 7f413d3ea640 1 -- 172.21.15.112:0/3737570664 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41380708b0 con 0x7f413813e960 2024-03-29T10:57:20.988 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.005+0000 7f413d3ea640 10 monclient: _finish_auth 0 2024-03-29T10:57:20.988 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.005+0000 7f413d3ea640 10 monclient: _check_auth_tickets 2024-03-29T10:57:20.988 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.005+0000 7f413d3ea640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:20.988 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.005+0000 7f413d3ea640 1 -- 172.21.15.112:0/3737570664 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f4134001670 con 0x7f413813e960 2024-03-29T10:57:20.988 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.005+0000 7f413d3ea640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:20.988 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.005+0000 7f413d3ea640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:20.988 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.005+0000 7f413d3ea640 10 monclient: dump: 2024-03-29T10:57:20.988 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:20.988 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:20.988 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:20.988 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:20.989 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:20.989 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:20.989 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:20.989 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:20.989 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:20.989 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:20.989 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.005+0000 7f4140650640 5 monclient: authenticate success, global_id 4423 2024-03-29T10:57:20.989 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.005+0000 7f4140650640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:20.989 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.006+0000 7f413d3ea640 1 -- 172.21.15.112:0/3737570664 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41340017d0 con 0x7f413813e960 2024-03-29T10:57:20.989 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.006+0000 7f413d3ea640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:20.989 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.006+0000 7f413d3ea640 1 -- 172.21.15.112:0/3737570664 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f4134001be0 con 0x7f413813e960 2024-03-29T10:57:20.989 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.006+0000 7f413d3ea640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:20.989 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.006+0000 7f413d3ea640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:20.989 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.006+0000 7f413d3ea640 10 monclient: dump: 2024-03-29T10:57:20.989 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:20.990 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:20.990 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:20.990 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:20.990 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:20.990 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:20.990 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:20.990 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:20.990 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:20.990 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:20.990 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.006+0000 7f4140650640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:20.990 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.006+0000 7f4140650640 4 set_mon_vals no callback set 2024-03-29T10:57:20.991 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.006+0000 7f4140650640 10 monclient: shutdown 2024-03-29T10:57:20.991 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.006+0000 7f4140650640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:20.991 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.006+0000 7f4140650640 1 -- 172.21.15.112:0/3737570664 >> v1:172.21.15.118:6789/0 conn(0x7f413813e960 legacy=0x7f413806e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:20.991 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.010+0000 7f4140650640 1 -- 172.21.15.112:0/3737570664 shutdown_connections 2024-03-29T10:57:20.991 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.010+0000 7f4140650640 1 -- 172.21.15.112:0/3737570664 >> 172.21.15.112:0/3737570664 conn(0x7f4138067570 msgr2=0x7f41380b3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:20.994 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.010+0000 7f4140650640 1 -- 172.21.15.112:0/3737570664 shutdown_connections 2024-03-29T10:57:20.994 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.011+0000 7f4140650640 1 -- 172.21.15.112:0/3737570664 wait complete. 2024-03-29T10:57:20.994 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.012+0000 7f4140650640 10 monclient: build_initial_monmap 2024-03-29T10:57:20.994 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.012+0000 7f4140650640 10 monclient: monmap: 2024-03-29T10:57:20.994 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:57:20.994 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:20.994 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:57:20.994 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:57:20.994 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:20.994 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:20.995 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:57:20.995 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:57:20.995 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:57:20.995 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:20.995 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.012+0000 7f4140650640 1 librados: starting msgr at 2024-03-29T10:57:20.995 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.012+0000 7f4140650640 1 librados: starting objecter 2024-03-29T10:57:20.995 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.013+0000 7f4140650640 1 Processor -- start 2024-03-29T10:57:20.995 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.013+0000 7f4140650640 1 -- start start 2024-03-29T10:57:20.995 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.013+0000 7f4140650640 1 librados: setting wanted keys 2024-03-29T10:57:20.995 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.013+0000 7f4140650640 1 librados: calling monclient init 2024-03-29T10:57:20.996 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.013+0000 7f4140650640 10 monclient: init 2024-03-29T10:57:20.996 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.013+0000 7f4140650640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:20.996 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.013+0000 7f4140650640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:57:20.996 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.013+0000 7f4140650640 10 monclient(hunting): picked mon.noname-a con 0x7f41380683f0 addr v1:172.21.15.112:6789/0 2024-03-29T10:57:20.996 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.013+0000 7f4140650640 10 monclient(hunting): picked mon.noname-c con 0x7f413813b030 addr v1:172.21.15.112:6790/0 2024-03-29T10:57:20.996 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.013+0000 7f4140650640 10 monclient(hunting): picked mon.noname-b con 0x7f413813e960 addr v1:172.21.15.118:6789/0 2024-03-29T10:57:20.996 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.013+0000 7f4140650640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4138140650 con 0x7f41380683f0 2024-03-29T10:57:20.996 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.013+0000 7f4140650640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4138141180 con 0x7f413813e960 2024-03-29T10:57:20.996 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.013+0000 7f4140650640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413813d3b0 con 0x7f413813b030 2024-03-29T10:57:20.996 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.013+0000 7f4140650640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:20.996 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.013+0000 7f4140650640 10 monclient(hunting): authenticate will time out at 1196.319171s 2024-03-29T10:57:20.998 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.017+0000 7f413dbeb640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f413813b030 0x7f4138140a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:47208/0 (socket says 172.21.15.112:47208) 2024-03-29T10:57:20.998 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.017+0000 7f413dbeb640 1 -- 172.21.15.112:0/1045321460 learned_addr learned my addr 172.21.15.112:0/1045321460 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:57:20.999 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.017+0000 7f412effd640 10 client.?.objecter ms_handle_connect 0x7f413813b030 2024-03-29T10:57:20.999 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.017+0000 7f412effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:20.999 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.017+0000 7f412effd640 10 client.?.objecter ms_handle_connect 0x7f41380683f0 2024-03-29T10:57:20.999 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.017+0000 7f412effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:20.999 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.017+0000 7f412effd640 1 -- 172.21.15.112:0/1045321460 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17362631 0 0) 0x7f413813d3b0 con 0x7f413813b030 2024-03-29T10:57:20.999 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.017+0000 7f412effd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:20.999 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.017+0000 7f412effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:20.999 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.017+0000 7f412effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:20.999 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.017+0000 7f412effd640 10 monclient(hunting): my global_id is 4460 2024-03-29T10:57:20.999 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.017+0000 7f412effd640 1 -- 172.21.15.112:0/1045321460 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f411c004eb0 con 0x7f413813b030 2024-03-29T10:57:21.002 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.018+0000 7f412effd640 1 -- 172.21.15.112:0/1045321460 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300636781 0 0) 0x7f4138140650 con 0x7f41380683f0 2024-03-29T10:57:21.003 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.018+0000 7f412effd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:21.003 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.018+0000 7f412effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:21.003 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.018+0000 7f412effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:21.003 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.018+0000 7f412effd640 10 monclient(hunting): my global_id is 4434 2024-03-29T10:57:21.003 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.018+0000 7f412effd640 1 -- 172.21.15.112:0/1045321460 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f413813d3b0 con 0x7f41380683f0 2024-03-29T10:57:21.003 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.018+0000 7f412effd640 1 -- 172.21.15.112:0/1045321460 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 85844908 0 0) 0x7f411c004eb0 con 0x7f413813b030 2024-03-29T10:57:21.003 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.018+0000 7f412effd640 1 -- 172.21.15.112:0/1045321460 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f411c0064f0 con 0x7f413813b030 2024-03-29T10:57:21.003 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f412effd640 10 client.?.objecter ms_handle_connect 0x7f413813e960 2024-03-29T10:57:21.003 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f412effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:21.003 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f412effd640 1 -- 172.21.15.112:0/1045321460 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316938303 0 0) 0x7f4138141180 con 0x7f413813e960 2024-03-29T10:57:21.003 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f412effd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:21.003 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f412effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:21.003 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f412effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:21.003 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f412effd640 10 monclient(hunting): my global_id is 4426 2024-03-29T10:57:21.004 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f412effd640 1 -- 172.21.15.112:0/1045321460 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f411c004eb0 con 0x7f413813e960 2024-03-29T10:57:21.004 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f412effd640 1 -- 172.21.15.112:0/1045321460 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f4134001c20 con 0x7f413813b030 2024-03-29T10:57:21.004 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f412effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:21.004 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f412effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:21.004 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f412effd640 10 monclient(hunting): dump: 2024-03-29T10:57:21.004 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:21.004 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:21.004 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:21.004 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:21.004 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:21.004 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:21.004 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:21.004 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:21.004 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:21.004 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:21.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f412effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:21.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f412effd640 1 -- 172.21.15.112:0/1045321460 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2787157474 0 0) 0x7f411c0064f0 con 0x7f413813b030 2024-03-29T10:57:21.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f412effd640 1 -- 172.21.15.112:0/1045321460 >> v1:172.21.15.118:6789/0 conn(0x7f413813e960 legacy=0x7f41381e65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:21.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f412effd640 1 -- 172.21.15.112:0/1045321460 >> v1:172.21.15.112:6789/0 conn(0x7f41380683f0 legacy=0x7f413813d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:21.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f412effd640 10 monclient: _finish_hunting 0 2024-03-29T10:57:21.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f412effd640 1 monclient: found mon.c 2024-03-29T10:57:21.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f412effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:21.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f412effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:21.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f412effd640 1 -- 172.21.15.112:0/1045321460 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413813d540 con 0x7f413813b030 2024-03-29T10:57:21.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f412effd640 10 monclient: _finish_auth 0 2024-03-29T10:57:21.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f412effd640 10 monclient: _check_auth_tickets 2024-03-29T10:57:21.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f412effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:21.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f4140650640 5 monclient: authenticate success, global_id 4460 2024-03-29T10:57:21.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f4140650640 10 monclient: _renew_subs 2024-03-29T10:57:21.005 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f4140650640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:21.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f4140650640 1 -- 172.21.15.112:0/1045321460 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f413813f380 con 0x7f413813b030 2024-03-29T10:57:21.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f4140650640 10 client.4460.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:21.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f4140650640 10 monclient: _renew_subs 2024-03-29T10:57:21.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f4140650640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:21.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f4140650640 1 -- 172.21.15.112:0/1045321460 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4138151c30 con 0x7f413813b030 2024-03-29T10:57:21.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.019+0000 7f4140650640 1 librados: init done 2024-03-29T10:57:21.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.020+0000 7f412effd640 1 -- 172.21.15.112:0/1045321460 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41340038a0 con 0x7f413813b030 2024-03-29T10:57:21.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.020+0000 7f412effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:21.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.020+0000 7f413cbe9640 4 set_mon_vals no callback set 2024-03-29T10:57:21.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.020+0000 7f412effd640 1 -- 172.21.15.112:0/1045321460 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f4134003bf0 con 0x7f413813b030 2024-03-29T10:57:21.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.020+0000 7f412effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:21.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.020+0000 7f412effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:21.006 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.020+0000 7f412effd640 10 monclient: dump: 2024-03-29T10:57:21.006 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:21.006 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:21.007 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:21.007 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:21.007 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:21.007 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:21.007 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:21.007 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:21.007 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:21.007 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:21.008 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.023+0000 7f4140650640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:21.008 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.023+0000 7f4140650640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:21.008 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.023+0000 7f4140650640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:21.008 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.023+0000 7f412effd640 1 -- 172.21.15.112:0/1045321460 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f4134003dd0 con 0x7f413813b030 2024-03-29T10:57:21.008 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.024+0000 7f4140650640 1 -- 172.21.15.112:0/1045321460 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4120005080 con 0x7f413813b030 2024-03-29T10:57:21.009 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.024+0000 7f412effd640 1 -- 172.21.15.112:0/1045321460 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f4134054a20 con 0x7f413813b030 2024-03-29T10:57:21.009 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.024+0000 7f412effd640 10 client.4460.objecter ms_dispatch 0x7f4138000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:21.009 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.024+0000 7f412effd640 3 client.4460.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:21.009 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.024+0000 7f412effd640 3 client.4460.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:21.009 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.024+0000 7f412effd640 20 client.4460.objecter dump_active .. 0 homeless 2024-03-29T10:57:21.009 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.025+0000 7f413e3ec640 10 monclient: get_auth_request con 0x7f411c06df50 auth_method 0 2024-03-29T10:57:21.009 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.026+0000 7f412effd640 10 client.4460.objecter ms_handle_connect 0x7f411c06df50 2024-03-29T10:57:21.015 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.033+0000 7f412effd640 1 -- 172.21.15.112:0/1045321460 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4134050cc0 con 0x7f413813b030 2024-03-29T10:57:21.015 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.033+0000 7f412effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:21.015 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.033+0000 7f412effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:21.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.356+0000 7f4140650640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:21.338 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.356+0000 7f4140650640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:21.338 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.356+0000 7f4140650640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:21.338 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.356+0000 7f4140650640 1 -- 172.21.15.112:0/1045321460 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4120005330 con 0x7f413813b030 2024-03-29T10:57:21.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.359+0000 7f412effd640 1 -- 172.21.15.112:0/1045321460 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f4134050e60 con 0x7f413813b030 2024-03-29T10:57:21.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.359+0000 7f412effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:21.340 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.359+0000 7f412effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:21.341 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:57:21.341 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:21.344 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.362+0000 7f4140650640 10 librados: watch_flush enter 2024-03-29T10:57:21.344 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.362+0000 7f4140650640 10 librados: watch_flush exit 2024-03-29T10:57:21.344 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.362+0000 7f4140650640 20 client.4460.objecter shutdown clearing up homeless session... 2024-03-29T10:57:21.344 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.362+0000 7f4140650640 10 client.4460.objecter successfully canceled tick 2024-03-29T10:57:21.344 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.363+0000 7f4140650640 1 -- 172.21.15.112:0/1045321460 >> v1:172.21.15.118:6816/3181073408 conn(0x7f411c06df50 legacy=0x7f411c070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:21.344 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.363+0000 7f4140650640 10 monclient: shutdown 2024-03-29T10:57:21.344 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.363+0000 7f4140650640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:21.345 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.363+0000 7f4140650640 1 -- 172.21.15.112:0/1045321460 >> v1:172.21.15.112:6790/0 conn(0x7f413813b030 legacy=0x7f4138140a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:21.345 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.363+0000 7f4140650640 1 -- 172.21.15.112:0/1045321460 shutdown_connections 2024-03-29T10:57:21.345 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.363+0000 7f4140650640 1 -- 172.21.15.112:0/1045321460 >> 172.21.15.112:0/1045321460 conn(0x7f4138067570 msgr2=0x7f413806def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:21.345 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.363+0000 7f4140650640 1 -- 172.21.15.112:0/1045321460 shutdown_connections 2024-03-29T10:57:21.345 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.363+0000 7f4140650640 1 -- 172.21.15.112:0/1045321460 wait complete. 2024-03-29T10:57:21.346 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:21.364+0000 7f4140650640 1 librados: shutdown 2024-03-29T10:57:21.365 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:28.172 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:28.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.341+0000 7f82ba496640 10 monclient: get_monmap_and_config 2024-03-29T10:57:28.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.341+0000 7f82ba496640 10 monclient: build_initial_monmap 2024-03-29T10:57:28.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.341+0000 7f82ba496640 10 monclient: monmap: 2024-03-29T10:57:28.327 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:57:28.327 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:28.327 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:57:28.342727+0000 2024-03-29T10:57:28.327 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:57:28.342727+0000 2024-03-29T10:57:28.327 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:28.327 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:28.327 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:57:28.327 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:57:28.327 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:57:28.327 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:28.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.342+0000 7f82ba496640 1 Processor -- start 2024-03-29T10:57:28.327 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.343+0000 7f82ba496640 1 -- start start 2024-03-29T10:57:28.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.343+0000 7f82ba496640 10 monclient: init 2024-03-29T10:57:28.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.343+0000 7f82ba496640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:28.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.343+0000 7f82ba496640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:57:28.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.343+0000 7f82ba496640 10 monclient(hunting): picked mon.noname-c con 0x7f82b40a9750 addr v1:172.21.15.118:6789/0 2024-03-29T10:57:28.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.343+0000 7f82ba496640 10 monclient(hunting): picked mon.noname-a con 0x7f82b4149ac0 addr v1:172.21.15.112:6789/0 2024-03-29T10:57:28.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.343+0000 7f82ba496640 10 monclient(hunting): picked mon.noname-b con 0x7f82b414fe50 addr v1:172.21.15.112:6790/0 2024-03-29T10:57:28.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.343+0000 7f82ba496640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b40aa4a0 con 0x7f82b4149ac0 2024-03-29T10:57:28.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.343+0000 7f82ba496640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b414a680 con 0x7f82b40a9750 2024-03-29T10:57:28.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.343+0000 7f82ba496640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b40af290 con 0x7f82b414fe50 2024-03-29T10:57:28.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.343+0000 7f82ba496640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:28.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.343+0000 7f82ba496640 10 monclient(hunting): authenticate will time out at 1203.648090s 2024-03-29T10:57:28.328 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.345+0000 7f82b8a33640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f82b414fe50 0x7f82b4152240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:50214/0 (socket says 172.21.15.112:50214) 2024-03-29T10:57:28.329 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.345+0000 7f82b8a33640 1 -- 172.21.15.112:0/974738870 learned_addr learned my addr 172.21.15.112:0/974738870 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:57:28.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.349+0000 7f82b2ffd640 1 -- 172.21.15.112:0/974738870 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170948486 0 0) 0x7f82b414a680 con 0x7f82b40a9750 2024-03-29T10:57:28.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.349+0000 7f82b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:28.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.349+0000 7f82b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:28.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.349+0000 7f82b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:28.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.349+0000 7f82b2ffd640 10 monclient(hunting): my global_id is 4429 2024-03-29T10:57:28.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.349+0000 7f82b2ffd640 1 -- 172.21.15.112:0/974738870 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f829c003520 con 0x7f82b40a9750 2024-03-29T10:57:28.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.349+0000 7f82b2ffd640 1 -- 172.21.15.112:0/974738870 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999986250 0 0) 0x7f82b40af290 con 0x7f82b414fe50 2024-03-29T10:57:28.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.349+0000 7f82b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:28.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.349+0000 7f82b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:28.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.349+0000 7f82b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:28.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.349+0000 7f82b2ffd640 10 monclient(hunting): my global_id is 4463 2024-03-29T10:57:28.331 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.349+0000 7f82b2ffd640 1 -- 172.21.15.112:0/974738870 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b414a680 con 0x7f82b414fe50 2024-03-29T10:57:28.332 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.349+0000 7f82b2ffd640 1 -- 172.21.15.112:0/974738870 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149598360 0 0) 0x7f82b40aa4a0 con 0x7f82b4149ac0 2024-03-29T10:57:28.332 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.349+0000 7f82b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:28.332 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.349+0000 7f82b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:28.332 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.349+0000 7f82b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:28.332 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.349+0000 7f82b2ffd640 10 monclient(hunting): my global_id is 4437 2024-03-29T10:57:28.332 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.349+0000 7f82b2ffd640 1 -- 172.21.15.112:0/974738870 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b40af290 con 0x7f82b4149ac0 2024-03-29T10:57:28.332 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.349+0000 7f82b2ffd640 1 -- 172.21.15.112:0/974738870 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 684336307 0 0) 0x7f829c003520 con 0x7f82b40a9750 2024-03-29T10:57:28.332 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.349+0000 7f82b2ffd640 1 -- 172.21.15.112:0/974738870 >> v1:172.21.15.112:6790/0 conn(0x7f82b414fe50 legacy=0x7f82b4152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:28.332 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.349+0000 7f82b2ffd640 1 -- 172.21.15.112:0/974738870 >> v1:172.21.15.112:6789/0 conn(0x7f82b4149ac0 legacy=0x7f82b4149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:28.332 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.350+0000 7f82b2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:57:28.332 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.350+0000 7f82b2ffd640 1 monclient: found mon.noname-c 2024-03-29T10:57:28.332 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.350+0000 7f82b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:28.332 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.350+0000 7f82b2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T10:57:28.332 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.350+0000 7f82b2ffd640 1 -- 172.21.15.112:0/974738870 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b40afb40 con 0x7f82b40a9750 2024-03-29T10:57:28.333 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.350+0000 7f82b2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:57:28.333 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.350+0000 7f82b2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:57:28.333 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.350+0000 7f82b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:28.333 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.350+0000 7f82ba496640 5 monclient: authenticate success, global_id 4429 2024-03-29T10:57:28.333 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.350+0000 7f82ba496640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:28.333 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.350+0000 7f82b2ffd640 1 -- 172.21.15.112:0/974738870 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f82a4001690 con 0x7f82b40a9750 2024-03-29T10:57:28.333 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.350+0000 7f82b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:28.333 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.350+0000 7f82b2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:28.333 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.350+0000 7f82b2ffd640 10 monclient: dump: 2024-03-29T10:57:28.333 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:28.333 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:28.333 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:28.333 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:28.333 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:28.333 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:28.334 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:28.334 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:28.334 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:28.334 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:28.334 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.350+0000 7f82ba496640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:28.334 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.350+0000 7f82b2ffd640 1 -- 172.21.15.112:0/974738870 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82a40018b0 con 0x7f82b40a9750 2024-03-29T10:57:28.334 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.350+0000 7f82b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:28.334 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.350+0000 7f82ba496640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:28.334 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.350+0000 7f82ba496640 4 set_mon_vals no callback set 2024-03-29T10:57:28.334 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.350+0000 7f82b2ffd640 1 -- 172.21.15.112:0/974738870 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f82a4001c40 con 0x7f82b40a9750 2024-03-29T10:57:28.334 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.351+0000 7f82b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:28.334 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.351+0000 7f82b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:28.334 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.351+0000 7f82b2ffd640 10 monclient: dump: 2024-03-29T10:57:28.334 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:28.334 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:28.335 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:28.335 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:28.335 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:28.335 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:28.335 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:28.335 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:28.335 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:28.335 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:28.336 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.351+0000 7f82ba496640 10 monclient: shutdown 2024-03-29T10:57:28.336 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.351+0000 7f82ba496640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:28.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.351+0000 7f82ba496640 1 -- 172.21.15.112:0/974738870 >> v1:172.21.15.118:6789/0 conn(0x7f82b40a9750 legacy=0x7f82b41436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:28.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.351+0000 7f82ba496640 1 -- 172.21.15.112:0/974738870 shutdown_connections 2024-03-29T10:57:28.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.351+0000 7f82ba496640 1 -- 172.21.15.112:0/974738870 >> 172.21.15.112:0/974738870 conn(0x7f82b40b35e0 msgr2=0x7f82b40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:28.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.351+0000 7f82ba496640 1 -- 172.21.15.112:0/974738870 shutdown_connections 2024-03-29T10:57:28.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.351+0000 7f82ba496640 1 -- 172.21.15.112:0/974738870 wait complete. 2024-03-29T10:57:28.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.352+0000 7f82ba496640 10 monclient: build_initial_monmap 2024-03-29T10:57:28.337 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.352+0000 7f82ba496640 10 monclient: monmap: 2024-03-29T10:57:28.337 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:57:28.337 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:28.337 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:57:28.337 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:57:28.337 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:28.337 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:28.337 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:57:28.337 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:57:28.338 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:57:28.338 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:28.338 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.352+0000 7f82ba496640 1 librados: starting msgr at 2024-03-29T10:57:28.338 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.352+0000 7f82ba496640 1 librados: starting objecter 2024-03-29T10:57:28.338 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.352+0000 7f82ba496640 1 Processor -- start 2024-03-29T10:57:28.338 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.353+0000 7f82ba496640 1 -- start start 2024-03-29T10:57:28.338 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.353+0000 7f82ba496640 1 librados: setting wanted keys 2024-03-29T10:57:28.338 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.353+0000 7f82ba496640 1 librados: calling monclient init 2024-03-29T10:57:28.338 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.353+0000 7f82ba496640 10 monclient: init 2024-03-29T10:57:28.338 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.353+0000 7f82ba496640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:28.338 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.353+0000 7f82ba496640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:57:28.338 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.353+0000 7f82ba496640 10 monclient(hunting): picked mon.noname-a con 0x7f82b40a9750 addr v1:172.21.15.112:6789/0 2024-03-29T10:57:28.338 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.353+0000 7f82ba496640 10 monclient(hunting): picked mon.noname-b con 0x7f82b4149ac0 addr v1:172.21.15.118:6789/0 2024-03-29T10:57:28.338 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.354+0000 7f82ba496640 10 monclient(hunting): picked mon.noname-c con 0x7f82b414fe50 addr v1:172.21.15.112:6790/0 2024-03-29T10:57:28.338 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.354+0000 7f82ba496640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b4072490 con 0x7f82b40a9750 2024-03-29T10:57:28.339 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.354+0000 7f82ba496640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b4070fc0 con 0x7f82b4149ac0 2024-03-29T10:57:28.339 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.354+0000 7f82ba496640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b4070510 con 0x7f82b414fe50 2024-03-29T10:57:28.339 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.354+0000 7f82ba496640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:28.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.354+0000 7f82ba496640 10 monclient(hunting): authenticate will time out at 1203.658236s 2024-03-29T10:57:28.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.357+0000 7f82b3fff640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f82b40a9750 0x7f82b4140440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60513/0 (socket says 172.21.15.112:60513) 2024-03-29T10:57:28.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.357+0000 7f82b3fff640 1 -- 172.21.15.112:0/2409314977 learned_addr learned my addr 172.21.15.112:0/2409314977 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:57:28.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.358+0000 7f82b0ff9640 10 client.?.objecter ms_handle_connect 0x7f82b414fe50 2024-03-29T10:57:28.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.358+0000 7f82b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:28.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.358+0000 7f82b0ff9640 10 client.?.objecter ms_handle_connect 0x7f82b40a9750 2024-03-29T10:57:28.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.358+0000 7f82b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:28.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.358+0000 7f82b0ff9640 10 client.?.objecter ms_handle_connect 0x7f82b4149ac0 2024-03-29T10:57:28.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.358+0000 7f82b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:28.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.358+0000 7f82b0ff9640 1 -- 172.21.15.112:0/2409314977 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2030939876 0 0) 0x7f82b4072490 con 0x7f82b40a9750 2024-03-29T10:57:28.341 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.358+0000 7f82b0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:57:28.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.358+0000 7f82b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:28.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.358+0000 7f82b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:28.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.358+0000 7f82b0ff9640 10 monclient(hunting): my global_id is 4440 2024-03-29T10:57:28.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.358+0000 7f82b0ff9640 1 -- 172.21.15.112:0/2409314977 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82a0004eb0 con 0x7f82b40a9750 2024-03-29T10:57:28.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.358+0000 7f82b0ff9640 1 -- 172.21.15.112:0/2409314977 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737763008 0 0) 0x7f82b4070510 con 0x7f82b414fe50 2024-03-29T10:57:28.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.358+0000 7f82b0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:57:28.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.358+0000 7f82b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:28.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.358+0000 7f82b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:28.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.358+0000 7f82b0ff9640 10 monclient(hunting): my global_id is 4466 2024-03-29T10:57:28.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.358+0000 7f82b0ff9640 1 -- 172.21.15.112:0/2409314977 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82a0006230 con 0x7f82b414fe50 2024-03-29T10:57:28.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.358+0000 7f82b0ff9640 1 -- 172.21.15.112:0/2409314977 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3136382927 0 0) 0x7f82b4070fc0 con 0x7f82b4149ac0 2024-03-29T10:57:28.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.358+0000 7f82b0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:57:28.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.358+0000 7f82b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:28.342 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.358+0000 7f82b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:28.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.358+0000 7f82b0ff9640 10 monclient(hunting): my global_id is 4432 2024-03-29T10:57:28.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.358+0000 7f82b0ff9640 1 -- 172.21.15.112:0/2409314977 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b4070510 con 0x7f82b4149ac0 2024-03-29T10:57:28.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 1 -- 172.21.15.112:0/2409314977 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3965979286 0 0) 0x7f82a0006230 con 0x7f82b414fe50 2024-03-29T10:57:28.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 1 -- 172.21.15.112:0/2409314977 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82b4070fc0 con 0x7f82b414fe50 2024-03-29T10:57:28.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 1 -- 172.21.15.112:0/2409314977 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f82ac001c30 con 0x7f82b414fe50 2024-03-29T10:57:28.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:28.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:28.343 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 10 monclient(hunting): dump: 2024-03-29T10:57:28.343 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:28.343 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:28.343 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:28.343 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:28.343 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:28.343 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:28.343 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:28.344 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:28.344 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:28.344 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:28.344 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:28.344 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 1 -- 172.21.15.112:0/2409314977 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3850279153 0 0) 0x7f82a0004eb0 con 0x7f82b40a9750 2024-03-29T10:57:28.344 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 1 -- 172.21.15.112:0/2409314977 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82a0006d10 con 0x7f82b40a9750 2024-03-29T10:57:28.344 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 1 -- 172.21.15.112:0/2409314977 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f82a4001c00 con 0x7f82b40a9750 2024-03-29T10:57:28.344 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:28.344 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:28.344 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 10 monclient(hunting): dump: 2024-03-29T10:57:28.344 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:28.344 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:28.344 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:28.344 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:28.344 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:28.345 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:28.345 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:28.345 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:28.345 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:28.345 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:28.345 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 1 -- 172.21.15.112:0/2409314977 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2526261668 0 0) 0x7f82b4070510 con 0x7f82b4149ac0 2024-03-29T10:57:28.345 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 1 -- 172.21.15.112:0/2409314977 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82a0005320 con 0x7f82b4149ac0 2024-03-29T10:57:28.345 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 1 -- 172.21.15.112:0/2409314977 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 44711264 0 0) 0x7f82b4070fc0 con 0x7f82b414fe50 2024-03-29T10:57:28.345 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 1 -- 172.21.15.112:0/2409314977 >> v1:172.21.15.118:6789/0 conn(0x7f82b4149ac0 legacy=0x7f82b41ded10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:28.345 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 1 -- 172.21.15.112:0/2409314977 >> v1:172.21.15.112:6789/0 conn(0x7f82b40a9750 legacy=0x7f82b4140440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:28.345 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:57:28.345 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 1 monclient: found mon.c 2024-03-29T10:57:28.345 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:28.345 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:28.345 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 1 -- 172.21.15.112:0/2409314977 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b40706a0 con 0x7f82b414fe50 2024-03-29T10:57:28.346 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 10 monclient: _finish_auth 0 2024-03-29T10:57:28.346 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:57:28.346 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:28.346 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82ba496640 5 monclient: authenticate success, global_id 4466 2024-03-29T10:57:28.346 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82ba496640 10 monclient: _renew_subs 2024-03-29T10:57:28.346 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82ba496640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:28.346 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82ba496640 1 -- 172.21.15.112:0/2409314977 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b4146420 con 0x7f82b414fe50 2024-03-29T10:57:28.346 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82ba496640 10 client.4466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:28.346 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82ba496640 10 monclient: _renew_subs 2024-03-29T10:57:28.346 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82ba496640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:28.346 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82ba496640 1 -- 172.21.15.112:0/2409314977 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b4140b50 con 0x7f82b414fe50 2024-03-29T10:57:28.346 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.359+0000 7f82ba496640 1 librados: init done 2024-03-29T10:57:28.348 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.361+0000 7f82b0ff9640 1 -- 172.21.15.112:0/2409314977 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82ac0020c0 con 0x7f82b414fe50 2024-03-29T10:57:28.348 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.361+0000 7f82b0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:28.348 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.361+0000 7f82b0ff9640 1 -- 172.21.15.112:0/2409314977 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f82ac001c30 con 0x7f82b414fe50 2024-03-29T10:57:28.348 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.361+0000 7f82b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:28.348 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.361+0000 7f82b0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:28.348 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.361+0000 7f82b0ff9640 10 monclient: dump: 2024-03-29T10:57:28.348 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:28.348 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:28.348 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:28.348 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:28.349 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:28.349 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:28.349 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:28.349 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:28.349 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:28.349 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:28.349 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.361+0000 7f82b27fc640 4 set_mon_vals no callback set 2024-03-29T10:57:28.349 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.362+0000 7f82b0ff9640 1 -- 172.21.15.112:0/2409314977 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f82ac01a9d0 con 0x7f82b414fe50 2024-03-29T10:57:28.349 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.363+0000 7f82ba496640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:28.349 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.363+0000 7f82ba496640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:28.349 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.363+0000 7f82ba496640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:28.349 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.363+0000 7f82ba496640 1 -- 172.21.15.112:0/2409314977 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8294005080 con 0x7f82b414fe50 2024-03-29T10:57:28.349 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.363+0000 7f82b3fff640 10 monclient: get_auth_request con 0x7f82a006ea40 auth_method 0 2024-03-29T10:57:28.349 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.363+0000 7f82b0ff9640 1 -- 172.21.15.112:0/2409314977 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f82ac054550 con 0x7f82b414fe50 2024-03-29T10:57:28.349 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.363+0000 7f82b0ff9640 10 client.4466.objecter ms_dispatch 0x7f82b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:28.350 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.363+0000 7f82b0ff9640 3 client.4466.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:28.350 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.363+0000 7f82b0ff9640 3 client.4466.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:28.350 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.363+0000 7f82b0ff9640 20 client.4466.objecter dump_active .. 0 homeless 2024-03-29T10:57:28.350 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.363+0000 7f82b0ff9640 10 client.4466.objecter ms_handle_connect 0x7f82a006ea40 2024-03-29T10:57:28.352 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.372+0000 7f82b0ff9640 1 -- 172.21.15.112:0/2409314977 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f82ac0507f0 con 0x7f82b414fe50 2024-03-29T10:57:28.352 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.372+0000 7f82b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:28.352 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.372+0000 7f82b0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:28.677 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.696+0000 7f82ba496640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:28.677 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.696+0000 7f82ba496640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:28.678 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.696+0000 7f82ba496640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:28.678 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.696+0000 7f82ba496640 1 -- 172.21.15.112:0/2409314977 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8294005330 con 0x7f82b414fe50 2024-03-29T10:57:28.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.699+0000 7f82b0ff9640 1 -- 172.21.15.112:0/2409314977 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f82ac050990 con 0x7f82b414fe50 2024-03-29T10:57:28.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.699+0000 7f82b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:28.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.699+0000 7f82b0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:28.680 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:57:28.680 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:28.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.703+0000 7f82ba496640 10 librados: watch_flush enter 2024-03-29T10:57:28.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.703+0000 7f82ba496640 10 librados: watch_flush exit 2024-03-29T10:57:28.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.703+0000 7f82ba496640 20 client.4466.objecter shutdown clearing up homeless session... 2024-03-29T10:57:28.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.703+0000 7f82ba496640 10 client.4466.objecter successfully canceled tick 2024-03-29T10:57:28.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.703+0000 7f82ba496640 1 -- 172.21.15.112:0/2409314977 >> v1:172.21.15.118:6816/3181073408 conn(0x7f82a006ea40 legacy=0x7f82a0070f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:28.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.703+0000 7f82ba496640 10 monclient: shutdown 2024-03-29T10:57:28.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.703+0000 7f82ba496640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:28.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.703+0000 7f82ba496640 1 -- 172.21.15.112:0/2409314977 >> v1:172.21.15.112:6790/0 conn(0x7f82b414fe50 legacy=0x7f82b41df420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:28.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.703+0000 7f82ba496640 1 -- 172.21.15.112:0/2409314977 shutdown_connections 2024-03-29T10:57:28.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.703+0000 7f82ba496640 1 -- 172.21.15.112:0/2409314977 >> 172.21.15.112:0/2409314977 conn(0x7f82b40b35e0 msgr2=0x7f82b4142b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:28.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.704+0000 7f82ba496640 1 -- 172.21.15.112:0/2409314977 shutdown_connections 2024-03-29T10:57:28.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.704+0000 7f82ba496640 1 -- 172.21.15.112:0/2409314977 wait complete. 2024-03-29T10:57:28.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:28.705+0000 7f82ba496640 1 librados: shutdown 2024-03-29T10:57:28.701 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:35.508 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:35.660 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.679+0000 7f9343606640 10 monclient: get_monmap_and_config 2024-03-29T10:57:35.660 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.679+0000 7f9343606640 10 monclient: build_initial_monmap 2024-03-29T10:57:35.660 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.679+0000 7f9343606640 10 monclient: monmap: 2024-03-29T10:57:35.660 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:57:35.661 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:35.661 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:57:35.680828+0000 2024-03-29T10:57:35.661 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:57:35.680828+0000 2024-03-29T10:57:35.661 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:35.661 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:35.661 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:57:35.661 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:57:35.661 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:57:35.661 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:35.662 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.680+0000 7f9343606640 1 Processor -- start 2024-03-29T10:57:35.662 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.681+0000 7f9343606640 1 -- start start 2024-03-29T10:57:35.662 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.681+0000 7f9343606640 10 monclient: init 2024-03-29T10:57:35.662 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.681+0000 7f9343606640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:35.662 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.681+0000 7f9343606640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:35.662 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.681+0000 7f9343606640 10 monclient(hunting): picked mon.noname-c con 0x7f933c0b4920 addr v1:172.21.15.118:6789/0 2024-03-29T10:57:35.662 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.681+0000 7f9343606640 10 monclient(hunting): picked mon.noname-b con 0x7f933c0b1790 addr v1:172.21.15.112:6790/0 2024-03-29T10:57:35.662 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.682+0000 7f9343606640 10 monclient(hunting): picked mon.noname-a con 0x7f933c146790 addr v1:172.21.15.112:6789/0 2024-03-29T10:57:35.662 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.682+0000 7f9343606640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933c0b4070 con 0x7f933c146790 2024-03-29T10:57:35.663 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.682+0000 7f9340ba1640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f933c0b1790 0x7f933c0b1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:46612/0 (socket says 172.21.15.112:46612) 2024-03-29T10:57:35.663 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.682+0000 7f9340ba1640 1 -- 172.21.15.112:0/2421690701 learned_addr learned my addr 172.21.15.112:0/2421690701 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:57:35.663 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.682+0000 7f9343606640 1 -- 172.21.15.112:0/2421690701 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933c06fb80 con 0x7f933c0b4920 2024-03-29T10:57:35.663 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.682+0000 7f9343606640 1 -- 172.21.15.112:0/2421690701 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933c0b42c0 con 0x7f933c0b1790 2024-03-29T10:57:35.663 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.682+0000 7f9343606640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:35.663 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.682+0000 7f9343606640 10 monclient(hunting): authenticate will time out at 1210.986327s 2024-03-29T10:57:35.664 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.682+0000 7f9333fff640 1 -- 172.21.15.112:0/2421690701 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312166104 0 0) 0x7f933c0b42c0 con 0x7f933c0b1790 2024-03-29T10:57:35.664 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.682+0000 7f9333fff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:35.664 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.682+0000 7f9333fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:35.664 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.682+0000 7f9333fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:35.664 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.682+0000 7f9333fff640 10 monclient(hunting): my global_id is 4469 2024-03-29T10:57:35.664 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.682+0000 7f9333fff640 1 -- 172.21.15.112:0/2421690701 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9328003520 con 0x7f933c0b1790 2024-03-29T10:57:35.664 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 1 -- 172.21.15.112:0/2421690701 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3923877008 0 0) 0x7f933c0b4070 con 0x7f933c146790 2024-03-29T10:57:35.664 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:35.664 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:35.664 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:35.664 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 10 monclient(hunting): my global_id is 4443 2024-03-29T10:57:35.664 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 1 -- 172.21.15.112:0/2421690701 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f933c0b42c0 con 0x7f933c146790 2024-03-29T10:57:35.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 1 -- 172.21.15.112:0/2421690701 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583882991 0 0) 0x7f933c06fb80 con 0x7f933c0b4920 2024-03-29T10:57:35.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:35.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:35.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:35.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 10 monclient(hunting): my global_id is 4435 2024-03-29T10:57:35.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 1 -- 172.21.15.112:0/2421690701 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f933c0b4070 con 0x7f933c0b4920 2024-03-29T10:57:35.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 1 -- 172.21.15.112:0/2421690701 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4055948271 0 0) 0x7f933c0b42c0 con 0x7f933c146790 2024-03-29T10:57:35.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 1 -- 172.21.15.112:0/2421690701 >> v1:172.21.15.112:6790/0 conn(0x7f933c0b1790 legacy=0x7f933c0b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:35.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 1 -- 172.21.15.112:0/2421690701 >> v1:172.21.15.118:6789/0 conn(0x7f933c0b4920 legacy=0x7f933c06e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:35.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 10 monclient: _finish_hunting 0 2024-03-29T10:57:35.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 1 monclient: found mon.noname-a 2024-03-29T10:57:35.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:35.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T10:57:35.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 1 -- 172.21.15.112:0/2421690701 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933c0b22c0 con 0x7f933c146790 2024-03-29T10:57:35.666 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 10 monclient: _finish_auth 0 2024-03-29T10:57:35.666 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 10 monclient: _check_auth_tickets 2024-03-29T10:57:35.666 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:35.666 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 1 -- 172.21.15.112:0/2421690701 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9334001970 con 0x7f933c146790 2024-03-29T10:57:35.666 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:35.666 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:35.666 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9333fff640 10 monclient: dump: 2024-03-29T10:57:35.666 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:35.666 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:35.666 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:35.666 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:35.666 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:35.666 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:35.666 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:35.666 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:35.667 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:35.667 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:35.667 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9343606640 5 monclient: authenticate success, global_id 4443 2024-03-29T10:57:35.667 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.683+0000 7f9343606640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:35.675 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.684+0000 7f9333fff640 1 -- 172.21.15.112:0/2421690701 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9334001ad0 con 0x7f933c146790 2024-03-29T10:57:35.675 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.684+0000 7f9333fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:35.675 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.684+0000 7f9333fff640 1 -- 172.21.15.112:0/2421690701 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9334001ee0 con 0x7f933c146790 2024-03-29T10:57:35.676 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.684+0000 7f9333fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:35.676 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.684+0000 7f9333fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:35.676 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.684+0000 7f9333fff640 10 monclient: dump: 2024-03-29T10:57:35.676 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:35.676 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:35.676 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:35.676 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:35.676 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:35.676 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:35.676 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:35.676 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:35.676 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:35.676 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:35.676 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.684+0000 7f9343606640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:35.676 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.684+0000 7f9343606640 4 set_mon_vals no callback set 2024-03-29T10:57:35.677 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.685+0000 7f9343606640 10 monclient: shutdown 2024-03-29T10:57:35.677 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.685+0000 7f9343606640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:35.677 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.685+0000 7f9343606640 1 -- 172.21.15.112:0/2421690701 >> v1:172.21.15.112:6789/0 conn(0x7f933c146790 legacy=0x7f933c146c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:35.677 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.685+0000 7f9343606640 1 -- 172.21.15.112:0/2421690701 shutdown_connections 2024-03-29T10:57:35.677 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.685+0000 7f9343606640 1 -- 172.21.15.112:0/2421690701 >> 172.21.15.112:0/2421690701 conn(0x7f933c066a00 msgr2=0x7f933c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:35.677 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.685+0000 7f9343606640 1 -- 172.21.15.112:0/2421690701 shutdown_connections 2024-03-29T10:57:35.677 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.685+0000 7f9343606640 1 -- 172.21.15.112:0/2421690701 wait complete. 2024-03-29T10:57:35.677 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.686+0000 7f9343606640 10 monclient: build_initial_monmap 2024-03-29T10:57:35.677 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.686+0000 7f9343606640 10 monclient: monmap: 2024-03-29T10:57:35.677 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:57:35.677 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:35.677 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:57:35.677 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:57:35.677 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:35.678 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:35.678 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:57:35.678 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:57:35.678 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:57:35.678 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:35.678 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.687+0000 7f9343606640 1 librados: starting msgr at 2024-03-29T10:57:35.678 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.687+0000 7f9343606640 1 librados: starting objecter 2024-03-29T10:57:35.678 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.687+0000 7f9343606640 1 Processor -- start 2024-03-29T10:57:35.678 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.687+0000 7f9343606640 1 -- start start 2024-03-29T10:57:35.678 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.687+0000 7f9343606640 1 librados: setting wanted keys 2024-03-29T10:57:35.678 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.687+0000 7f9343606640 1 librados: calling monclient init 2024-03-29T10:57:35.678 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.687+0000 7f9343606640 10 monclient: init 2024-03-29T10:57:35.678 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.688+0000 7f9343606640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:35.678 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.688+0000 7f9343606640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:57:35.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.688+0000 7f9343606640 10 monclient(hunting): picked mon.noname-a con 0x7f933c0b1790 addr v1:172.21.15.112:6789/0 2024-03-29T10:57:35.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.688+0000 7f9343606640 10 monclient(hunting): picked mon.noname-c con 0x7f933c0b4920 addr v1:172.21.15.112:6790/0 2024-03-29T10:57:35.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.688+0000 7f9343606640 10 monclient(hunting): picked mon.noname-b con 0x7f933c146790 addr v1:172.21.15.118:6789/0 2024-03-29T10:57:35.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.688+0000 7f9343606640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933c0aba60 con 0x7f933c0b1790 2024-03-29T10:57:35.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.689+0000 7f9340ba1640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f933c0b4920 0x7f933c0aae20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:47334/0 (socket says 172.21.15.112:47334) 2024-03-29T10:57:35.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.689+0000 7f9340ba1640 1 -- 172.21.15.112:0/255561107 learned_addr learned my addr 172.21.15.112:0/255561107 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:57:35.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.689+0000 7f9331ffb640 10 client.?.objecter ms_handle_connect 0x7f933c0b4920 2024-03-29T10:57:35.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.689+0000 7f9331ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:35.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.689+0000 7f9331ffb640 10 client.?.objecter ms_handle_connect 0x7f933c146790 2024-03-29T10:57:35.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.689+0000 7f9331ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:35.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.689+0000 7f9343606640 1 -- 172.21.15.112:0/255561107 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933c1d9050 con 0x7f933c146790 2024-03-29T10:57:35.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.689+0000 7f9343606640 1 -- 172.21.15.112:0/255561107 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933c1db7d0 con 0x7f933c0b4920 2024-03-29T10:57:35.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.689+0000 7f9331ffb640 10 client.?.objecter ms_handle_connect 0x7f933c0b1790 2024-03-29T10:57:35.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.689+0000 7f9331ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:35.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.689+0000 7f9343606640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:35.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9343606640 10 monclient(hunting): authenticate will time out at 1210.994220s 2024-03-29T10:57:35.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 227744111 0 0) 0x7f933c0aba60 con 0x7f933c0b1790 2024-03-29T10:57:35.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:35.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:35.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:35.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 10 monclient(hunting): my global_id is 4446 2024-03-29T10:57:35.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9320004eb0 con 0x7f933c0b1790 2024-03-29T10:57:35.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656439294 0 0) 0x7f933c1d9050 con 0x7f933c146790 2024-03-29T10:57:35.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:35.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:35.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:35.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 10 monclient(hunting): my global_id is 4438 2024-03-29T10:57:35.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f933c0aba60 con 0x7f933c146790 2024-03-29T10:57:35.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517410620 0 0) 0x7f933c1db7d0 con 0x7f933c0b4920 2024-03-29T10:57:35.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:35.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:35.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:35.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 10 monclient(hunting): my global_id is 4472 2024-03-29T10:57:35.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f933c1d9050 con 0x7f933c0b4920 2024-03-29T10:57:35.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 44020595 0 0) 0x7f9320004eb0 con 0x7f933c0b1790 2024-03-29T10:57:35.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f933c1db7d0 con 0x7f933c0b1790 2024-03-29T10:57:35.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f93380018f0 con 0x7f933c0b1790 2024-03-29T10:57:35.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:35.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:35.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 10 monclient(hunting): dump: 2024-03-29T10:57:35.681 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:35.681 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:35.682 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:35.682 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:35.682 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:35.682 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:35.682 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:35.682 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:35.682 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:35.682 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:35.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.690+0000 7f9331ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:35.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3859541099 0 0) 0x7f933c1d9050 con 0x7f933c0b4920 2024-03-29T10:57:35.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9320004eb0 con 0x7f933c0b4920 2024-03-29T10:57:35.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 998225295 0 0) 0x7f933c0aba60 con 0x7f933c146790 2024-03-29T10:57:35.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f933c1d9050 con 0x7f933c146790 2024-03-29T10:57:35.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f932c001370 con 0x7f933c0b4920 2024-03-29T10:57:35.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:35.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:35.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 10 monclient(hunting): dump: 2024-03-29T10:57:35.683 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:35.683 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:35.683 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:35.683 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:35.683 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:35.683 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:35.683 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:35.683 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:35.683 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:35.683 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:35.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9334001de0 con 0x7f933c146790 2024-03-29T10:57:35.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:35.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:35.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 10 monclient(hunting): dump: 2024-03-29T10:57:35.684 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:35.684 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:35.684 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:35.684 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:35.684 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:35.684 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:35.684 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:35.684 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:35.684 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:35.684 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:35.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4041221002 0 0) 0x7f933c1db7d0 con 0x7f933c0b1790 2024-03-29T10:57:35.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 >> v1:172.21.15.112:6790/0 conn(0x7f933c0b4920 legacy=0x7f933c0aae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:35.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 >> v1:172.21.15.118:6789/0 conn(0x7f933c146790 legacy=0x7f933c1e5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:35.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:57:35.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 1 monclient: found mon.a 2024-03-29T10:57:35.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:35.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:57:35.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933c1db3f0 con 0x7f933c0b1790 2024-03-29T10:57:35.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 10 monclient: _finish_auth 0 2024-03-29T10:57:35.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:57:35.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:35.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9343606640 5 monclient: authenticate success, global_id 4446 2024-03-29T10:57:35.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9343606640 10 monclient: _renew_subs 2024-03-29T10:57:35.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9343606640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:57:35.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9343606640 1 -- 172.21.15.112:0/255561107 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f933c0ab660 con 0x7f933c0b1790 2024-03-29T10:57:35.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9338003610 con 0x7f933c0b1790 2024-03-29T10:57:35.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:35.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f93380039e0 con 0x7f933c0b1790 2024-03-29T10:57:35.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:35.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:35.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9331ffb640 10 monclient: dump: 2024-03-29T10:57:35.686 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:35.686 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:35.686 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:35.686 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:35.686 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:35.686 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:35.686 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:35.686 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:35.686 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:35.687 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:35.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f93337fe640 4 set_mon_vals no callback set 2024-03-29T10:57:35.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9343606640 10 client.4446.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:35.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9343606640 10 monclient: _renew_subs 2024-03-29T10:57:35.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.691+0000 7f9343606640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:57:35.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.692+0000 7f9343606640 1 -- 172.21.15.112:0/255561107 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f933c0aa560 con 0x7f933c0b1790 2024-03-29T10:57:35.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.692+0000 7f9343606640 1 librados: init done 2024-03-29T10:57:35.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.693+0000 7f9343606640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:35.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.693+0000 7f9343606640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:35.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.693+0000 7f9343606640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:57:35.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.693+0000 7f9343606640 1 -- 172.21.15.112:0/255561107 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f933c1db5e0 con 0x7f933c0b1790 2024-03-29T10:57:35.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.694+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f9338003b40 con 0x7f933c0b1790 2024-03-29T10:57:35.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.695+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f9338083880 con 0x7f933c0b1790 2024-03-29T10:57:35.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.695+0000 7f9331ffb640 10 client.4446.objecter ms_dispatch 0x7f933c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:35.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.695+0000 7f9331ffb640 3 client.4446.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:35.687 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.695+0000 7f9331ffb640 3 client.4446.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:35.688 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.695+0000 7f9331ffb640 20 client.4446.objecter dump_active .. 0 homeless 2024-03-29T10:57:35.688 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.695+0000 7f9340ba1640 10 monclient: get_auth_request con 0x7f932006e270 auth_method 0 2024-03-29T10:57:35.688 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.695+0000 7f9331ffb640 10 client.4446.objecter ms_handle_connect 0x7f932006e270 2024-03-29T10:57:35.688 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.703+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f933804f160 con 0x7f933c0b1790 2024-03-29T10:57:35.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.703+0000 7f9331ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:35.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:35.703+0000 7f9331ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:36.037 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:36.056+0000 7f9343606640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:36.037 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:36.056+0000 7f9343606640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:36.037 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:36.056+0000 7f9343606640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:57:36.037 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:36.056+0000 7f9343606640 1 -- 172.21.15.112:0/255561107 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f933c075f10 con 0x7f933c0b1790 2024-03-29T10:57:36.038 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:36.057+0000 7f9331ffb640 1 -- 172.21.15.112:0/255561107 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f9338051090 con 0x7f933c0b1790 2024-03-29T10:57:36.038 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:36.057+0000 7f9331ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:36.038 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:36.057+0000 7f9331ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:36.038 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:57:36.038 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:36.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:36.061+0000 7f9343606640 10 librados: watch_flush enter 2024-03-29T10:57:36.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:36.061+0000 7f9343606640 10 librados: watch_flush exit 2024-03-29T10:57:36.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:36.061+0000 7f9343606640 20 client.4446.objecter shutdown clearing up homeless session... 2024-03-29T10:57:36.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:36.061+0000 7f9343606640 10 client.4446.objecter successfully canceled tick 2024-03-29T10:57:36.042 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:36.061+0000 7f9343606640 1 -- 172.21.15.112:0/255561107 >> v1:172.21.15.118:6816/3181073408 conn(0x7f932006e270 legacy=0x7f9320070730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:36.042 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:36.061+0000 7f9343606640 10 monclient: shutdown 2024-03-29T10:57:36.042 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:36.061+0000 7f9343606640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:36.042 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:36.061+0000 7f9343606640 1 -- 172.21.15.112:0/255561107 >> v1:172.21.15.112:6789/0 conn(0x7f933c0b1790 legacy=0x7f933c1dbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:36.042 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:36.062+0000 7f9343606640 1 -- 172.21.15.112:0/255561107 shutdown_connections 2024-03-29T10:57:36.043 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:36.062+0000 7f9343606640 1 -- 172.21.15.112:0/255561107 >> 172.21.15.112:0/255561107 conn(0x7f933c066a00 msgr2=0x7f933c06e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:36.043 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:36.062+0000 7f9343606640 1 -- 172.21.15.112:0/255561107 shutdown_connections 2024-03-29T10:57:36.043 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:36.062+0000 7f9343606640 1 -- 172.21.15.112:0/255561107 wait complete. 2024-03-29T10:57:36.043 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:36.062+0000 7f9343606640 1 librados: shutdown 2024-03-29T10:57:36.056 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:42.862 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:43.030 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.049+0000 7fe9bd782640 10 monclient: get_monmap_and_config 2024-03-29T10:57:43.030 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.049+0000 7fe9bd782640 10 monclient: build_initial_monmap 2024-03-29T10:57:43.031 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.049+0000 7fe9bd782640 10 monclient: monmap: 2024-03-29T10:57:43.031 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:57:43.031 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:43.031 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:57:43.050024+0000 2024-03-29T10:57:43.031 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:57:43.050024+0000 2024-03-29T10:57:43.031 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:43.031 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:43.031 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:57:43.031 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:57:43.031 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:57:43.031 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:43.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.051+0000 7fe9bd782640 1 Processor -- start 2024-03-29T10:57:43.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.051+0000 7fe9bd782640 1 -- start start 2024-03-29T10:57:43.032 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.051+0000 7fe9bd782640 10 monclient: init 2024-03-29T10:57:43.034 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.052+0000 7fe9bd782640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:43.034 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.052+0000 7fe9bd782640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:57:43.039 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.055+0000 7fe9bd782640 10 monclient(hunting): picked mon.noname-a con 0x7fe9b80ae250 addr v1:172.21.15.112:6789/0 2024-03-29T10:57:43.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.055+0000 7fe9bd782640 10 monclient(hunting): picked mon.noname-c con 0x7fe9b813f970 addr v1:172.21.15.118:6789/0 2024-03-29T10:57:43.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.055+0000 7fe9bd782640 10 monclient(hunting): picked mon.noname-b con 0x7fe9b8142800 addr v1:172.21.15.112:6790/0 2024-03-29T10:57:43.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.058+0000 7fe9bd782640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b80afe00 con 0x7fe9b80ae250 2024-03-29T10:57:43.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.058+0000 7fe9bd782640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b8142c10 con 0x7fe9b813f970 2024-03-29T10:57:43.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.058+0000 7fe9bd782640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b80afb20 con 0x7fe9b8142800 2024-03-29T10:57:43.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.058+0000 7fe9bd782640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:43.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.058+0000 7fe9bd782640 10 monclient(hunting): authenticate will time out at 1218.362280s 2024-03-29T10:57:43.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.058+0000 7fe9b77fe640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe9b8142800 0x7fe9b814da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:39754/0 (socket says 172.21.15.112:39754) 2024-03-29T10:57:43.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.059+0000 7fe9b77fe640 1 -- 172.21.15.112:0/2935245900 learned_addr learned my addr 172.21.15.112:0/2935245900 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:57:43.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.059+0000 7fe9b67fc640 1 -- 172.21.15.112:0/2935245900 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120443228 0 0) 0x7fe9b80afb20 con 0x7fe9b8142800 2024-03-29T10:57:43.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.059+0000 7fe9b67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:43.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.059+0000 7fe9b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:43.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.059+0000 7fe9b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:43.040 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.059+0000 7fe9b67fc640 10 monclient(hunting): my global_id is 4475 2024-03-29T10:57:43.041 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.059+0000 7fe9b67fc640 1 -- 172.21.15.112:0/2935245900 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9a4003520 con 0x7fe9b8142800 2024-03-29T10:57:43.044 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 1 -- 172.21.15.112:0/2935245900 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170967785 0 0) 0x7fe9b80afe00 con 0x7fe9b80ae250 2024-03-29T10:57:43.044 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:43.045 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:43.045 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:43.045 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 10 monclient(hunting): my global_id is 4449 2024-03-29T10:57:43.045 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 1 -- 172.21.15.112:0/2935245900 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9b80afb20 con 0x7fe9b80ae250 2024-03-29T10:57:43.045 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 1 -- 172.21.15.112:0/2935245900 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3108068615 0 0) 0x7fe9b8142c10 con 0x7fe9b813f970 2024-03-29T10:57:43.045 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:43.045 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:43.045 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:43.045 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 10 monclient(hunting): my global_id is 4441 2024-03-29T10:57:43.045 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 1 -- 172.21.15.112:0/2935245900 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9b80afe00 con 0x7fe9b813f970 2024-03-29T10:57:43.045 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 1 -- 172.21.15.112:0/2935245900 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2614970305 0 0) 0x7fe9a4003520 con 0x7fe9b8142800 2024-03-29T10:57:43.045 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 1 -- 172.21.15.112:0/2935245900 >> v1:172.21.15.118:6789/0 conn(0x7fe9b813f970 legacy=0x7fe9b81420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:43.045 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 1 -- 172.21.15.112:0/2935245900 >> v1:172.21.15.112:6789/0 conn(0x7fe9b80ae250 legacy=0x7fe9b813f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:43.045 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:57:43.045 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 1 monclient: found mon.noname-b 2024-03-29T10:57:43.046 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:43.046 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:57:43.046 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 1 -- 172.21.15.112:0/2935245900 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b80aab80 con 0x7fe9b8142800 2024-03-29T10:57:43.046 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 10 monclient: _finish_auth 0 2024-03-29T10:57:43.046 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:57:43.046 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:43.046 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 1 -- 172.21.15.112:0/2935245900 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe9b0001990 con 0x7fe9b8142800 2024-03-29T10:57:43.046 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:43.046 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:43.046 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9b67fc640 10 monclient: dump: 2024-03-29T10:57:43.046 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:43.046 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:43.046 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:43.047 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:43.047 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:43.047 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:43.047 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:43.047 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:43.047 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:43.047 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:43.047 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9bd782640 5 monclient: authenticate success, global_id 4475 2024-03-29T10:57:43.047 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.064+0000 7fe9bd782640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:43.048 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.065+0000 7fe9b67fc640 1 -- 172.21.15.112:0/2935245900 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9b0001af0 con 0x7fe9b8142800 2024-03-29T10:57:43.048 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.065+0000 7fe9b67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:43.048 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.065+0000 7fe9b67fc640 1 -- 172.21.15.112:0/2935245900 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe9b0001f00 con 0x7fe9b8142800 2024-03-29T10:57:43.048 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.065+0000 7fe9bd782640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:43.048 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.065+0000 7fe9bd782640 4 set_mon_vals no callback set 2024-03-29T10:57:43.048 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.065+0000 7fe9b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:43.048 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.065+0000 7fe9b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:43.048 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.065+0000 7fe9b67fc640 10 monclient: dump: 2024-03-29T10:57:43.048 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:43.048 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:43.049 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:43.049 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:43.049 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:43.049 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:43.049 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:43.049 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:43.049 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:43.049 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:43.049 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.066+0000 7fe9bd782640 10 monclient: shutdown 2024-03-29T10:57:43.049 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.066+0000 7fe9bd782640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:43.049 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.066+0000 7fe9bd782640 1 -- 172.21.15.112:0/2935245900 >> v1:172.21.15.112:6790/0 conn(0x7fe9b8142800 legacy=0x7fe9b814da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:43.049 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.067+0000 7fe9bd782640 1 -- 172.21.15.112:0/2935245900 shutdown_connections 2024-03-29T10:57:43.049 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.067+0000 7fe9bd782640 1 -- 172.21.15.112:0/2935245900 >> 172.21.15.112:0/2935245900 conn(0x7fe9b80b35e0 msgr2=0x7fe9b80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:43.049 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.067+0000 7fe9bd782640 1 -- 172.21.15.112:0/2935245900 shutdown_connections 2024-03-29T10:57:43.051 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.067+0000 7fe9bd782640 1 -- 172.21.15.112:0/2935245900 wait complete. 2024-03-29T10:57:43.051 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.068+0000 7fe9bd782640 10 monclient: build_initial_monmap 2024-03-29T10:57:43.051 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.068+0000 7fe9bd782640 10 monclient: monmap: 2024-03-29T10:57:43.051 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:57:43.051 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:43.051 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:57:43.051 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:57:43.051 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:43.051 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:43.052 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:57:43.052 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:57:43.052 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:57:43.052 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:43.052 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.069+0000 7fe9bd782640 1 librados: starting msgr at 2024-03-29T10:57:43.052 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.069+0000 7fe9bd782640 1 librados: starting objecter 2024-03-29T10:57:43.052 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.069+0000 7fe9bd782640 1 Processor -- start 2024-03-29T10:57:43.052 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.069+0000 7fe9bd782640 1 -- start start 2024-03-29T10:57:43.052 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.069+0000 7fe9bd782640 1 librados: setting wanted keys 2024-03-29T10:57:43.052 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.069+0000 7fe9bd782640 1 librados: calling monclient init 2024-03-29T10:57:43.052 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.069+0000 7fe9bd782640 10 monclient: init 2024-03-29T10:57:43.052 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.070+0000 7fe9bd782640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:43.052 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.070+0000 7fe9bd782640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:43.053 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.070+0000 7fe9bd782640 10 monclient(hunting): picked mon.noname-c con 0x7fe9b80ae250 addr v1:172.21.15.112:6790/0 2024-03-29T10:57:43.053 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.072+0000 7fe9bd782640 10 monclient(hunting): picked mon.noname-a con 0x7fe9b813f970 addr v1:172.21.15.112:6789/0 2024-03-29T10:57:43.053 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.072+0000 7fe9bd782640 10 monclient(hunting): picked mon.noname-b con 0x7fe9b8142800 addr v1:172.21.15.118:6789/0 2024-03-29T10:57:43.053 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.072+0000 7fe9bd782640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b8145b70 con 0x7fe9b813f970 2024-03-29T10:57:43.053 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.072+0000 7fe9bd782640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b8145dc0 con 0x7fe9b8142800 2024-03-29T10:57:43.053 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.072+0000 7fe9bd782640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b8138410 con 0x7fe9b80ae250 2024-03-29T10:57:43.053 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.072+0000 7fe9bd782640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:43.053 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.072+0000 7fe9bd782640 10 monclient(hunting): authenticate will time out at 1218.376481s 2024-03-29T10:57:43.057 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.076+0000 7fe9b6ffd640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe9b80ae250 0x7fe9b8144500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:42986/0 (socket says 172.21.15.112:42986) 2024-03-29T10:57:43.057 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.076+0000 7fe9b6ffd640 1 -- 172.21.15.112:0/1984030972 learned_addr learned my addr 172.21.15.112:0/1984030972 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:57:43.058 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.078+0000 7fe9af7fe640 10 client.?.objecter ms_handle_connect 0x7fe9b8142800 2024-03-29T10:57:43.058 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.078+0000 7fe9af7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:43.058 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.078+0000 7fe9af7fe640 10 client.?.objecter ms_handle_connect 0x7fe9b813f970 2024-03-29T10:57:43.060 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.078+0000 7fe9af7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:43.060 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.079+0000 7fe9af7fe640 1 -- 172.21.15.112:0/1984030972 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156558187 0 0) 0x7fe9b8145dc0 con 0x7fe9b8142800 2024-03-29T10:57:43.060 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.079+0000 7fe9af7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:43.060 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.079+0000 7fe9af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:43.060 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.079+0000 7fe9af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:43.060 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.079+0000 7fe9af7fe640 10 monclient(hunting): my global_id is 4444 2024-03-29T10:57:43.060 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.079+0000 7fe9af7fe640 1 -- 172.21.15.112:0/1984030972 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe998004eb0 con 0x7fe9b8142800 2024-03-29T10:57:43.060 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.079+0000 7fe9af7fe640 1 -- 172.21.15.112:0/1984030972 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217433507 0 0) 0x7fe9b8145b70 con 0x7fe9b813f970 2024-03-29T10:57:43.060 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.079+0000 7fe9af7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:43.060 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.079+0000 7fe9af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:43.060 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.079+0000 7fe9af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:43.060 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.079+0000 7fe9af7fe640 10 monclient(hunting): my global_id is 4452 2024-03-29T10:57:43.060 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.079+0000 7fe9af7fe640 1 -- 172.21.15.112:0/1984030972 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9b8145dc0 con 0x7fe9b813f970 2024-03-29T10:57:43.061 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.079+0000 7fe9af7fe640 1 -- 172.21.15.112:0/1984030972 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2918957660 0 0) 0x7fe9b8145dc0 con 0x7fe9b813f970 2024-03-29T10:57:43.061 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.079+0000 7fe9af7fe640 1 -- 172.21.15.112:0/1984030972 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9b8145b70 con 0x7fe9b813f970 2024-03-29T10:57:43.061 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.079+0000 7fe9af7fe640 1 -- 172.21.15.112:0/1984030972 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe9a0001830 con 0x7fe9b813f970 2024-03-29T10:57:43.061 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.079+0000 7fe9af7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:43.061 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.079+0000 7fe9af7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:43.061 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.079+0000 7fe9af7fe640 10 monclient(hunting): dump: 2024-03-29T10:57:43.061 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:43.061 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:43.061 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:43.061 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:43.061 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:43.061 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:43.061 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:43.061 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:43.061 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:43.062 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:43.062 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.079+0000 7fe9af7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:43.062 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.079+0000 7fe9af7fe640 10 client.?.objecter ms_handle_connect 0x7fe9b80ae250 2024-03-29T10:57:43.062 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.079+0000 7fe9af7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:43.064 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.079+0000 7fe9af7fe640 1 -- 172.21.15.112:0/1984030972 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1356048434 0 0) 0x7fe9b8145b70 con 0x7fe9b813f970 2024-03-29T10:57:43.064 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.079+0000 7fe9af7fe640 1 -- 172.21.15.112:0/1984030972 >> v1:172.21.15.112:6790/0 conn(0x7fe9b80ae250 legacy=0x7fe9b8144500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:43.064 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.080+0000 7fe9af7fe640 1 -- 172.21.15.112:0/1984030972 >> v1:172.21.15.118:6789/0 conn(0x7fe9b8142800 legacy=0x7fe9b81e5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:43.064 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.080+0000 7fe9af7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:57:43.064 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.080+0000 7fe9af7fe640 1 monclient: found mon.a 2024-03-29T10:57:43.064 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.080+0000 7fe9af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:43.064 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.080+0000 7fe9af7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:57:43.064 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.080+0000 7fe9af7fe640 1 -- 172.21.15.112:0/1984030972 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b811f200 con 0x7fe9b813f970 2024-03-29T10:57:43.064 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.080+0000 7fe9af7fe640 10 monclient: _finish_auth 0 2024-03-29T10:57:43.064 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.080+0000 7fe9af7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:57:43.064 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.080+0000 7fe9af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:43.064 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.080+0000 7fe9bd782640 5 monclient: authenticate success, global_id 4452 2024-03-29T10:57:43.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.080+0000 7fe9bd782640 10 monclient: _renew_subs 2024-03-29T10:57:43.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.080+0000 7fe9bd782640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:57:43.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.080+0000 7fe9bd782640 1 -- 172.21.15.112:0/1984030972 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9b813a920 con 0x7fe9b813f970 2024-03-29T10:57:43.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.080+0000 7fe9bd782640 10 client.4452.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:43.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.080+0000 7fe9bd782640 10 monclient: _renew_subs 2024-03-29T10:57:43.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.080+0000 7fe9bd782640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:57:43.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.080+0000 7fe9bd782640 1 -- 172.21.15.112:0/1984030972 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b81463c0 con 0x7fe9b813f970 2024-03-29T10:57:43.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.080+0000 7fe9bd782640 1 librados: init done 2024-03-29T10:57:43.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.081+0000 7fe9bd782640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:43.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.081+0000 7fe9bd782640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:43.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.081+0000 7fe9bd782640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:57:43.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.081+0000 7fe9bd782640 1 -- 172.21.15.112:0/1984030972 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9b8075f10 con 0x7fe9b813f970 2024-03-29T10:57:43.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.082+0000 7fe9af7fe640 1 -- 172.21.15.112:0/1984030972 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9a0001990 con 0x7fe9b813f970 2024-03-29T10:57:43.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.083+0000 7fe9af7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:43.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.083+0000 7fe9af7fe640 1 -- 172.21.15.112:0/1984030972 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe9a00037b0 con 0x7fe9b813f970 2024-03-29T10:57:43.066 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.083+0000 7fe9af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:43.066 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.083+0000 7fe9af7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:43.066 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.083+0000 7fe9af7fe640 10 monclient: dump: 2024-03-29T10:57:43.066 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:43.066 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:43.066 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:43.066 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:43.066 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:43.066 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:43.066 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:43.066 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:43.066 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:43.066 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:43.067 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.084+0000 7fe9b5ffb640 4 set_mon_vals no callback set 2024-03-29T10:57:43.067 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.084+0000 7fe9af7fe640 1 -- 172.21.15.112:0/1984030972 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fe9a0019840 con 0x7fe9b813f970 2024-03-29T10:57:43.067 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.085+0000 7fe9af7fe640 1 -- 172.21.15.112:0/1984030972 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7fe9a00837e0 con 0x7fe9b813f970 2024-03-29T10:57:43.067 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.085+0000 7fe9af7fe640 10 client.4452.objecter ms_dispatch 0x7fe9b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:43.067 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.085+0000 7fe9af7fe640 3 client.4452.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:43.068 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.085+0000 7fe9af7fe640 3 client.4452.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:43.068 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.085+0000 7fe9af7fe640 20 client.4452.objecter dump_active .. 0 homeless 2024-03-29T10:57:43.068 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.086+0000 7fe9b6ffd640 10 monclient: get_auth_request con 0x7fe99806ddd0 auth_method 0 2024-03-29T10:57:43.068 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.086+0000 7fe9af7fe640 10 client.4452.objecter ms_handle_connect 0x7fe99806ddd0 2024-03-29T10:57:43.074 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.092+0000 7fe9af7fe640 1 -- 172.21.15.112:0/1984030972 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9a004eef0 con 0x7fe9b813f970 2024-03-29T10:57:43.074 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.092+0000 7fe9af7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:43.074 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.092+0000 7fe9af7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:43.464 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.483+0000 7fe9bd782640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:43.464 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.483+0000 7fe9bd782640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:43.464 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.483+0000 7fe9bd782640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:57:43.464 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.483+0000 7fe9bd782640 1 -- 172.21.15.112:0/1984030972 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9b81385a0 con 0x7fe9b813f970 2024-03-29T10:57:43.464 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.484+0000 7fe9af7fe640 1 -- 172.21.15.112:0/1984030972 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7fe9a0050ff0 con 0x7fe9b813f970 2024-03-29T10:57:43.464 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.484+0000 7fe9af7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:43.464 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.484+0000 7fe9af7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:43.465 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:57:43.465 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:43.468 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.488+0000 7fe9bd782640 10 librados: watch_flush enter 2024-03-29T10:57:43.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.488+0000 7fe9bd782640 10 librados: watch_flush exit 2024-03-29T10:57:43.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.488+0000 7fe9bd782640 20 client.4452.objecter shutdown clearing up homeless session... 2024-03-29T10:57:43.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.488+0000 7fe9bd782640 10 client.4452.objecter successfully canceled tick 2024-03-29T10:57:43.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.488+0000 7fe9bd782640 1 -- 172.21.15.112:0/1984030972 >> v1:172.21.15.118:6816/3181073408 conn(0x7fe99806ddd0 legacy=0x7fe998070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:43.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.488+0000 7fe9bd782640 10 monclient: shutdown 2024-03-29T10:57:43.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.488+0000 7fe9bd782640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:43.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.488+0000 7fe9bd782640 1 -- 172.21.15.112:0/1984030972 >> v1:172.21.15.112:6789/0 conn(0x7fe9b813f970 legacy=0x7fe9b813a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:43.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.488+0000 7fe9bd782640 1 -- 172.21.15.112:0/1984030972 shutdown_connections 2024-03-29T10:57:43.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.488+0000 7fe9bd782640 1 -- 172.21.15.112:0/1984030972 >> 172.21.15.112:0/1984030972 conn(0x7fe9b80b35e0 msgr2=0x7fe9b813e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:43.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.488+0000 7fe9bd782640 1 -- 172.21.15.112:0/1984030972 shutdown_connections 2024-03-29T10:57:43.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.488+0000 7fe9bd782640 1 -- 172.21.15.112:0/1984030972 wait complete. 2024-03-29T10:57:43.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:43.488+0000 7fe9bd782640 1 librados: shutdown 2024-03-29T10:57:43.483 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:50.289 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:50.439 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.458+0000 7f9743eb3640 10 monclient: get_monmap_and_config 2024-03-29T10:57:50.439 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.458+0000 7f9743eb3640 10 monclient: build_initial_monmap 2024-03-29T10:57:50.439 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.458+0000 7f9743eb3640 10 monclient: monmap: 2024-03-29T10:57:50.439 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:57:50.439 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:50.439 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:57:50.459465+0000 2024-03-29T10:57:50.439 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:57:50.459465+0000 2024-03-29T10:57:50.439 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:50.439 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:50.440 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:57:50.440 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:57:50.440 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:57:50.440 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:50.440 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.459+0000 7f9743eb3640 1 Processor -- start 2024-03-29T10:57:50.440 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.459+0000 7f9743eb3640 1 -- start start 2024-03-29T10:57:50.440 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.459+0000 7f9743eb3640 10 monclient: init 2024-03-29T10:57:50.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.460+0000 7f9743eb3640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:50.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.460+0000 7f9743eb3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:50.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.460+0000 7f9743eb3640 10 monclient(hunting): picked mon.noname-c con 0x7f973c0b0770 addr v1:172.21.15.118:6789/0 2024-03-29T10:57:50.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.460+0000 7f9743eb3640 10 monclient(hunting): picked mon.noname-b con 0x7f973c14a700 addr v1:172.21.15.112:6790/0 2024-03-29T10:57:50.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.460+0000 7f9743eb3640 10 monclient(hunting): picked mon.noname-a con 0x7f973c0ac5a0 addr v1:172.21.15.112:6789/0 2024-03-29T10:57:50.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.460+0000 7f9743eb3640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f973c0aba70 con 0x7f973c0ac5a0 2024-03-29T10:57:50.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.460+0000 7f9743eb3640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f973c0aecb0 con 0x7f973c0b0770 2024-03-29T10:57:50.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.460+0000 7f9743eb3640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f973c0af150 con 0x7f973c14a700 2024-03-29T10:57:50.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.460+0000 7f9743eb3640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:50.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.460+0000 7f9743eb3640 10 monclient(hunting): authenticate will time out at 1225.765090s 2024-03-29T10:57:50.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.461+0000 7f9741c4f640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f973c0b0770 0x7f973c149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:44368/0 (socket says 172.21.15.112:44368) 2024-03-29T10:57:50.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.461+0000 7f9741c4f640 1 -- 172.21.15.112:0/1391156199 learned_addr learned my addr 172.21.15.112:0/1391156199 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:57:50.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.461+0000 7f9740c4d640 1 -- 172.21.15.112:0/1391156199 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351666315 0 0) 0x7f973c0af150 con 0x7f973c14a700 2024-03-29T10:57:50.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.461+0000 7f9740c4d640 10 monclient(hunting): _negotiate 2024-03-29T10:57:50.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.461+0000 7f9740c4d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:50.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.461+0000 7f9740c4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:50.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.461+0000 7f9740c4d640 10 monclient(hunting): my global_id is 4481 2024-03-29T10:57:50.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.461+0000 7f9740c4d640 1 -- 172.21.15.112:0/1391156199 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9728003520 con 0x7f973c14a700 2024-03-29T10:57:50.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.461+0000 7f9740c4d640 1 -- 172.21.15.112:0/1391156199 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866449584 0 0) 0x7f973c0aecb0 con 0x7f973c0b0770 2024-03-29T10:57:50.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.461+0000 7f9740c4d640 10 monclient(hunting): _negotiate 2024-03-29T10:57:50.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.461+0000 7f9740c4d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:50.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.461+0000 7f9740c4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:50.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.461+0000 7f9740c4d640 10 monclient(hunting): my global_id is 4447 2024-03-29T10:57:50.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.461+0000 7f9740c4d640 1 -- 172.21.15.112:0/1391156199 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f973c0af150 con 0x7f973c0b0770 2024-03-29T10:57:50.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.462+0000 7f9740c4d640 1 -- 172.21.15.112:0/1391156199 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118166844 0 0) 0x7f973c0aba70 con 0x7f973c0ac5a0 2024-03-29T10:57:50.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.462+0000 7f9740c4d640 10 monclient(hunting): _negotiate 2024-03-29T10:57:50.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.462+0000 7f9740c4d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:50.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.462+0000 7f9740c4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:50.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.462+0000 7f9740c4d640 10 monclient(hunting): my global_id is 4455 2024-03-29T10:57:50.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.462+0000 7f9740c4d640 1 -- 172.21.15.112:0/1391156199 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f973c0aecb0 con 0x7f973c0ac5a0 2024-03-29T10:57:50.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.462+0000 7f9740c4d640 1 -- 172.21.15.112:0/1391156199 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2059431571 0 0) 0x7f9728003520 con 0x7f973c14a700 2024-03-29T10:57:50.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.462+0000 7f9740c4d640 1 -- 172.21.15.112:0/1391156199 >> v1:172.21.15.118:6789/0 conn(0x7f973c0b0770 legacy=0x7f973c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:50.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.462+0000 7f9740c4d640 1 -- 172.21.15.112:0/1391156199 >> v1:172.21.15.112:6789/0 conn(0x7f973c0ac5a0 legacy=0x7f973c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:50.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.462+0000 7f9740c4d640 10 monclient: _finish_hunting 0 2024-03-29T10:57:50.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.462+0000 7f9740c4d640 1 monclient: found mon.noname-b 2024-03-29T10:57:50.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.462+0000 7f9740c4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:50.446 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.462+0000 7f9740c4d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:57:50.446 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.462+0000 7f9740c4d640 1 -- 172.21.15.112:0/1391156199 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973c0ac9d0 con 0x7f973c14a700 2024-03-29T10:57:50.446 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.462+0000 7f9740c4d640 10 monclient: _finish_auth 0 2024-03-29T10:57:50.446 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.462+0000 7f9740c4d640 10 monclient: _check_auth_tickets 2024-03-29T10:57:50.446 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.462+0000 7f9740c4d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:50.446 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.462+0000 7f9740c4d640 1 -- 172.21.15.112:0/1391156199 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f972c001690 con 0x7f973c14a700 2024-03-29T10:57:50.446 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.462+0000 7f9740c4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:50.446 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.462+0000 7f9740c4d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:50.446 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.462+0000 7f9740c4d640 10 monclient: dump: 2024-03-29T10:57:50.446 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:50.446 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:50.446 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:50.446 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:50.446 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:50.447 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:50.447 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:50.447 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:50.447 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:50.447 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:50.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.462+0000 7f9743eb3640 5 monclient: authenticate success, global_id 4481 2024-03-29T10:57:50.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.462+0000 7f9743eb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:50.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.463+0000 7f9740c4d640 1 -- 172.21.15.112:0/1391156199 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f972c001870 con 0x7f973c14a700 2024-03-29T10:57:50.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.463+0000 7f9740c4d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:50.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.463+0000 7f9743eb3640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:50.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.463+0000 7f9743eb3640 4 set_mon_vals no callback set 2024-03-29T10:57:50.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.463+0000 7f9740c4d640 1 -- 172.21.15.112:0/1391156199 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f972c001cc0 con 0x7f973c14a700 2024-03-29T10:57:50.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.463+0000 7f9740c4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:50.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.463+0000 7f9740c4d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:50.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.463+0000 7f9740c4d640 10 monclient: dump: 2024-03-29T10:57:50.448 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:50.448 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:50.448 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:50.448 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:50.448 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:50.448 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:50.448 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:50.448 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:50.448 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:50.448 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:50.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.464+0000 7f9743eb3640 10 monclient: shutdown 2024-03-29T10:57:50.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.464+0000 7f9743eb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:50.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.464+0000 7f9743eb3640 1 -- 172.21.15.112:0/1391156199 >> v1:172.21.15.112:6790/0 conn(0x7f973c14a700 legacy=0x7f973c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:50.449 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.464+0000 7f9743eb3640 1 -- 172.21.15.112:0/1391156199 shutdown_connections 2024-03-29T10:57:50.449 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.464+0000 7f9743eb3640 1 -- 172.21.15.112:0/1391156199 >> 172.21.15.112:0/1391156199 conn(0x7f973c06b660 msgr2=0x7f973c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:50.449 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.464+0000 7f9743eb3640 1 -- 172.21.15.112:0/1391156199 shutdown_connections 2024-03-29T10:57:50.449 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.464+0000 7f9743eb3640 1 -- 172.21.15.112:0/1391156199 wait complete. 2024-03-29T10:57:50.453 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.465+0000 7f9743eb3640 10 monclient: build_initial_monmap 2024-03-29T10:57:50.453 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.465+0000 7f9743eb3640 10 monclient: monmap: 2024-03-29T10:57:50.454 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:57:50.454 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:50.454 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:57:50.454 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:57:50.454 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:50.454 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:50.454 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:57:50.454 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:57:50.454 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:57:50.454 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:50.454 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.465+0000 7f9743eb3640 1 librados: starting msgr at 2024-03-29T10:57:50.454 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.465+0000 7f9743eb3640 1 librados: starting objecter 2024-03-29T10:57:50.455 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.465+0000 7f9743eb3640 1 Processor -- start 2024-03-29T10:57:50.455 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.465+0000 7f9743eb3640 1 -- start start 2024-03-29T10:57:50.455 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.466+0000 7f9743eb3640 1 librados: setting wanted keys 2024-03-29T10:57:50.455 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.466+0000 7f9743eb3640 1 librados: calling monclient init 2024-03-29T10:57:50.455 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.466+0000 7f9743eb3640 10 monclient: init 2024-03-29T10:57:50.455 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.466+0000 7f9743eb3640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:50.455 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.466+0000 7f9743eb3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:57:50.455 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.466+0000 7f9743eb3640 10 monclient(hunting): picked mon.noname-c con 0x7f973c0ac5a0 addr v1:172.21.15.112:6790/0 2024-03-29T10:57:50.455 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.466+0000 7f9743eb3640 10 monclient(hunting): picked mon.noname-b con 0x7f973c0b0770 addr v1:172.21.15.118:6789/0 2024-03-29T10:57:50.455 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.467+0000 7f9743eb3640 10 monclient(hunting): picked mon.noname-a con 0x7f973c14a700 addr v1:172.21.15.112:6789/0 2024-03-29T10:57:50.455 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.467+0000 7f9743eb3640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f973c145b30 con 0x7f973c14a700 2024-03-29T10:57:50.455 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.467+0000 7f9743eb3640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f973c128610 con 0x7f973c0b0770 2024-03-29T10:57:50.455 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.467+0000 7f9743eb3640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f973c1239d0 con 0x7f973c0ac5a0 2024-03-29T10:57:50.456 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.467+0000 7f9741c4f640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f973c0ac5a0 0x7f973c126af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:49192/0 (socket says 172.21.15.112:49192) 2024-03-29T10:57:50.456 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.467+0000 7f9741c4f640 1 -- 172.21.15.112:0/3510991743 learned_addr learned my addr 172.21.15.112:0/3510991743 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:57:50.456 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.467+0000 7f9743eb3640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:50.456 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.467+0000 7f9743eb3640 10 monclient(hunting): authenticate will time out at 1225.771294s 2024-03-29T10:57:50.456 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.467+0000 7f97327fc640 10 client.?.objecter ms_handle_connect 0x7f973c0ac5a0 2024-03-29T10:57:50.456 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.467+0000 7f97327fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:50.456 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.467+0000 7f97327fc640 1 -- 172.21.15.112:0/3510991743 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816126074 0 0) 0x7f973c1239d0 con 0x7f973c0ac5a0 2024-03-29T10:57:50.456 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.467+0000 7f97327fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:50.456 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.467+0000 7f97327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:50.456 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.467+0000 7f97327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:50.456 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.467+0000 7f97327fc640 10 monclient(hunting): my global_id is 4484 2024-03-29T10:57:50.456 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.467+0000 7f97327fc640 1 -- 172.21.15.112:0/3510991743 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9720004eb0 con 0x7f973c0ac5a0 2024-03-29T10:57:50.456 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.467+0000 7f97327fc640 10 client.?.objecter ms_handle_connect 0x7f973c14a700 2024-03-29T10:57:50.456 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.467+0000 7f97327fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:50.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 10 client.?.objecter ms_handle_connect 0x7f973c0b0770 2024-03-29T10:57:50.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:50.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 1 -- 172.21.15.112:0/3510991743 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021574664 0 0) 0x7f973c145b30 con 0x7f973c14a700 2024-03-29T10:57:50.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:50.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:50.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:50.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 10 monclient(hunting): my global_id is 4458 2024-03-29T10:57:50.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 1 -- 172.21.15.112:0/3510991743 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f973c1239d0 con 0x7f973c14a700 2024-03-29T10:57:50.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f9742450640 1 -- 172.21.15.112:0/3510991743 >> v1:172.21.15.112:6789/0 conn(0x7f973c14a700 legacy=0x7f973c145420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-29T10:57:50.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f9742450640 1 -- 172.21.15.112:0/3510991743 >> v1:172.21.15.112:6789/0 conn(0x7f973c14a700 legacy=0x7f973c145420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:57:50.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f9742450640 1 --1- 172.21.15.112:0/3510991743 >> v1:172.21.15.112:6789/0 conn(0x7f973c14a700 0x7f973c145420 :-1 s=OPENED pgs=664270 cs=1 l=1).handle_message read tag failed 2024-03-29T10:57:50.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f9742450640 1 --1- 172.21.15.112:0/3510991743 >> v1:172.21.15.112:6789/0 conn(0x7f973c14a700 0x7f973c145420 :-1 s=OPENED pgs=664270 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:57:50.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 1 -- 172.21.15.112:0/3510991743 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3135822468 0 0) 0x7f9720004eb0 con 0x7f973c0ac5a0 2024-03-29T10:57:50.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 1 -- 172.21.15.112:0/3510991743 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f973c145b30 con 0x7f973c0ac5a0 2024-03-29T10:57:50.458 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.112:6789/0 2024-03-29T10:57:50.458 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 1 -- 172.21.15.112:0/3510991743 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9738001ad0 con 0x7f973c0ac5a0 2024-03-29T10:57:50.458 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:50.458 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:50.458 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 10 monclient(hunting): dump: 2024-03-29T10:57:50.458 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:50.458 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:50.458 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:50.458 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:50.458 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:50.458 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:50.458 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:50.458 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:50.459 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:50.459 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:50.459 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:50.459 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 1 -- 172.21.15.112:0/3510991743 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1853043868 0 0) 0x7f973c128610 con 0x7f973c0b0770 2024-03-29T10:57:50.459 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:50.459 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:50.459 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:50.459 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 10 monclient(hunting): my global_id is 4450 2024-03-29T10:57:50.459 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 1 -- 172.21.15.112:0/3510991743 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9720004eb0 con 0x7f973c0b0770 2024-03-29T10:57:50.459 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 1 -- 172.21.15.112:0/3510991743 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3333790994 0 0) 0x7f973c145b30 con 0x7f973c0ac5a0 2024-03-29T10:57:50.459 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 1 -- 172.21.15.112:0/3510991743 >> v1:172.21.15.118:6789/0 conn(0x7f973c0b0770 legacy=0x7f973c144d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:50.459 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 1 -- 172.21.15.112:0/3510991743 >> v1:172.21.15.112:6789/0 conn(0x7f973c14a700 legacy=0x7f973c145420 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:57:50.459 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 10 monclient: _finish_hunting 0 2024-03-29T10:57:50.460 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 1 monclient: found mon.c 2024-03-29T10:57:50.460 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:50.460 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:50.460 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 1 -- 172.21.15.112:0/3510991743 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973c1dc350 con 0x7f973c0ac5a0 2024-03-29T10:57:50.460 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 10 monclient: _finish_auth 0 2024-03-29T10:57:50.460 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 10 monclient: _check_auth_tickets 2024-03-29T10:57:50.460 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f97327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:50.460 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f9743eb3640 5 monclient: authenticate success, global_id 4484 2024-03-29T10:57:50.460 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f9743eb3640 10 monclient: _renew_subs 2024-03-29T10:57:50.460 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f9743eb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:50.460 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f9743eb3640 1 -- 172.21.15.112:0/3510991743 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f973c1db910 con 0x7f973c0ac5a0 2024-03-29T10:57:50.460 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f9743eb3640 10 client.4484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:50.460 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f9743eb3640 10 monclient: _renew_subs 2024-03-29T10:57:50.460 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.468+0000 7f9743eb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:50.460 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.469+0000 7f9743eb3640 1 -- 172.21.15.112:0/3510991743 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f973c126190 con 0x7f973c0ac5a0 2024-03-29T10:57:50.461 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.469+0000 7f9743eb3640 1 librados: init done 2024-03-29T10:57:50.461 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.469+0000 7f97327fc640 1 -- 172.21.15.112:0/3510991743 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97380047e0 con 0x7f973c0ac5a0 2024-03-29T10:57:50.461 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.469+0000 7f97327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:50.461 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.469+0000 7f97337fe640 4 set_mon_vals no callback set 2024-03-29T10:57:50.461 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.470+0000 7f97327fc640 1 -- 172.21.15.112:0/3510991743 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9738004af0 con 0x7f973c0ac5a0 2024-03-29T10:57:50.461 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.470+0000 7f97327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:50.461 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.470+0000 7f97327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:50.461 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.470+0000 7f97327fc640 10 monclient: dump: 2024-03-29T10:57:50.461 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:50.461 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:50.461 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:50.461 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:50.461 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:50.461 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:50.462 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:50.462 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:50.462 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:50.462 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:50.462 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.470+0000 7f9743eb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:50.462 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.470+0000 7f9743eb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:50.462 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.471+0000 7f9743eb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:50.462 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.471+0000 7f9743eb3640 1 -- 172.21.15.112:0/3510991743 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9724005080 con 0x7f973c0ac5a0 2024-03-29T10:57:50.462 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.472+0000 7f97327fc640 1 -- 172.21.15.112:0/3510991743 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f973801aac0 con 0x7f973c0ac5a0 2024-03-29T10:57:50.462 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.472+0000 7f97327fc640 1 -- 172.21.15.112:0/3510991743 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f9738084a80 con 0x7f973c0ac5a0 2024-03-29T10:57:50.462 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.472+0000 7f97327fc640 10 client.4484.objecter ms_dispatch 0x7f973c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:50.462 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.472+0000 7f97327fc640 3 client.4484.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:50.462 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.472+0000 7f97327fc640 3 client.4484.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:50.462 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.472+0000 7f974144e640 10 monclient: get_auth_request con 0x7f972006de30 auth_method 0 2024-03-29T10:57:50.463 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.472+0000 7f97327fc640 20 client.4484.objecter dump_active .. 0 homeless 2024-03-29T10:57:50.463 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.473+0000 7f97327fc640 10 client.4484.objecter ms_handle_connect 0x7f972006de30 2024-03-29T10:57:50.463 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.481+0000 7f97327fc640 1 -- 172.21.15.112:0/3510991743 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9738050190 con 0x7f973c0ac5a0 2024-03-29T10:57:50.463 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.481+0000 7f97327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:50.463 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.481+0000 7f97327fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:50.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.787+0000 7f9743eb3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:50.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.787+0000 7f9743eb3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:50.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.787+0000 7f9743eb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:50.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.787+0000 7f9743eb3640 1 -- 172.21.15.112:0/3510991743 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9724005330 con 0x7f973c0ac5a0 2024-03-29T10:57:50.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.788+0000 7f97327fc640 1 -- 172.21.15.112:0/3510991743 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f9738052290 con 0x7f973c0ac5a0 2024-03-29T10:57:50.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.788+0000 7f97327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:50.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.788+0000 7f97327fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:50.768 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:57:50.769 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:50.771 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.790+0000 7f9743eb3640 10 librados: watch_flush enter 2024-03-29T10:57:50.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.791+0000 7f9743eb3640 10 librados: watch_flush exit 2024-03-29T10:57:50.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.791+0000 7f9743eb3640 20 client.4484.objecter shutdown clearing up homeless session... 2024-03-29T10:57:50.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.791+0000 7f9743eb3640 10 client.4484.objecter successfully canceled tick 2024-03-29T10:57:50.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.791+0000 7f9743eb3640 1 -- 172.21.15.112:0/3510991743 >> v1:172.21.15.118:6816/3181073408 conn(0x7f972006de30 legacy=0x7f97200702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:50.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.791+0000 7f9743eb3640 10 monclient: shutdown 2024-03-29T10:57:50.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.791+0000 7f9743eb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:50.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.791+0000 7f9743eb3640 1 -- 172.21.15.112:0/3510991743 >> v1:172.21.15.112:6790/0 conn(0x7f973c0ac5a0 legacy=0x7f973c126af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:50.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.791+0000 7f9743eb3640 1 -- 172.21.15.112:0/3510991743 shutdown_connections 2024-03-29T10:57:50.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.791+0000 7f9743eb3640 1 -- 172.21.15.112:0/3510991743 >> 172.21.15.112:0/3510991743 conn(0x7f973c06b660 msgr2=0x7f973c0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:50.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.791+0000 7f9743eb3640 1 -- 172.21.15.112:0/3510991743 shutdown_connections 2024-03-29T10:57:50.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.791+0000 7f9743eb3640 1 -- 172.21.15.112:0/3510991743 wait complete. 2024-03-29T10:57:50.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:50.792+0000 7f9743eb3640 1 librados: shutdown 2024-03-29T10:57:50.786 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:57.592 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:57.737 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.756+0000 7f7c7a4b3640 10 monclient: get_monmap_and_config 2024-03-29T10:57:57.737 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.756+0000 7f7c7a4b3640 10 monclient: build_initial_monmap 2024-03-29T10:57:57.737 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.756+0000 7f7c7a4b3640 10 monclient: monmap: 2024-03-29T10:57:57.737 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:57:57.737 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:57.737 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:57:57.757707+0000 2024-03-29T10:57:57.738 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:57:57.757707+0000 2024-03-29T10:57:57.738 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:57.738 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:57.738 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:57:57.738 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:57:57.738 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:57:57.738 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:57.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.757+0000 7f7c7a4b3640 1 Processor -- start 2024-03-29T10:57:57.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.757+0000 7f7c7a4b3640 1 -- start start 2024-03-29T10:57:57.739 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.757+0000 7f7c7a4b3640 10 monclient: init 2024-03-29T10:57:57.742 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.758+0000 7f7c7a4b3640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:57.742 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.758+0000 7f7c7a4b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:57.742 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.758+0000 7f7c7a4b3640 10 monclient(hunting): picked mon.noname-c con 0x7f7c740a9750 addr v1:172.21.15.118:6789/0 2024-03-29T10:57:57.742 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.758+0000 7f7c7a4b3640 10 monclient(hunting): picked mon.noname-b con 0x7f7c740aad20 addr v1:172.21.15.112:6790/0 2024-03-29T10:57:57.742 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.758+0000 7f7c7a4b3640 10 monclient(hunting): picked mon.noname-a con 0x7f7c740a9b30 addr v1:172.21.15.112:6789/0 2024-03-29T10:57:57.742 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.758+0000 7f7c7a4b3640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c740af4d0 con 0x7f7c740a9b30 2024-03-29T10:57:57.742 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.758+0000 7f7c7a4b3640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c740af7b0 con 0x7f7c740a9750 2024-03-29T10:57:57.742 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.758+0000 7f7c7a4b3640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c740acab0 con 0x7f7c740aad20 2024-03-29T10:57:57.742 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.758+0000 7f7c7a4b3640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:57.742 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.758+0000 7f7c7a4b3640 10 monclient(hunting): authenticate will time out at 1233.063072s 2024-03-29T10:57:57.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.766+0000 7f7c794b1640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f7c740a9750 0x7f7c740ac3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:37579/0 (socket says 172.21.15.112:37579) 2024-03-29T10:57:57.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.766+0000 7f7c794b1640 1 -- 172.21.15.112:0/2381534417 learned_addr learned my addr 172.21.15.112:0/2381534417 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:57:57.747 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.766+0000 7f7c6bfff640 1 -- 172.21.15.112:0/2381534417 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2481250436 0 0) 0x7f7c740af7b0 con 0x7f7c740a9750 2024-03-29T10:57:57.747 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.766+0000 7f7c6bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:57.747 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.766+0000 7f7c6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:57.747 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.766+0000 7f7c6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:57.747 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.766+0000 7f7c6bfff640 10 monclient(hunting): my global_id is 4453 2024-03-29T10:57:57.747 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.766+0000 7f7c6bfff640 1 -- 172.21.15.112:0/2381534417 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c60003520 con 0x7f7c740a9750 2024-03-29T10:57:57.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.769+0000 7f7c6bfff640 1 -- 172.21.15.112:0/2381534417 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2401781561 0 0) 0x7f7c60003520 con 0x7f7c740a9750 2024-03-29T10:57:57.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.769+0000 7f7c6bfff640 1 -- 172.21.15.112:0/2381534417 >> v1:172.21.15.112:6790/0 conn(0x7f7c740aad20 legacy=0x7f7c74141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:57:57.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.769+0000 7f7c6bfff640 1 -- 172.21.15.112:0/2381534417 >> v1:172.21.15.112:6789/0 conn(0x7f7c740a9b30 legacy=0x7f7c74149930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:57.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.769+0000 7f7c6bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:57:57.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.769+0000 7f7c6bfff640 1 monclient: found mon.noname-c 2024-03-29T10:57:57.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.769+0000 7f7c6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:57.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.769+0000 7f7c6bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T10:57:57.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.769+0000 7f7c6bfff640 1 -- 172.21.15.112:0/2381534417 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c740aba30 con 0x7f7c740a9750 2024-03-29T10:57:57.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.769+0000 7f7c6bfff640 10 monclient: _finish_auth 0 2024-03-29T10:57:57.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.769+0000 7f7c6bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:57:57.751 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.769+0000 7f7c6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:57.751 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.769+0000 7f7c6bfff640 1 -- 172.21.15.112:0/2381534417 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f7c64001690 con 0x7f7c740a9750 2024-03-29T10:57:57.751 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.769+0000 7f7c6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:57.751 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.769+0000 7f7c6bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:57.751 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.769+0000 7f7c6bfff640 10 monclient: dump: 2024-03-29T10:57:57.751 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:57.751 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:57.751 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:57.751 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:57.751 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:57.751 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:57.751 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:57.751 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:57.751 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:57.751 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:57.751 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.770+0000 7f7c6bfff640 1 -- 172.21.15.112:0/2381534417 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c640018b0 con 0x7f7c740a9750 2024-03-29T10:57:57.752 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.770+0000 7f7c6bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:57.752 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.770+0000 7f7c6bfff640 1 -- 172.21.15.112:0/2381534417 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f7c64001c40 con 0x7f7c740a9750 2024-03-29T10:57:57.752 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.770+0000 7f7c6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:57.752 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.770+0000 7f7c6bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:57.752 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.770+0000 7f7c6bfff640 10 monclient: dump: 2024-03-29T10:57:57.752 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:57.752 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:57.752 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:57.752 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:57.752 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:57.752 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:57.752 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:57.752 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:57.752 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:57.753 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:57.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.771+0000 7f7c7a4b3640 5 monclient: authenticate success, global_id 4453 2024-03-29T10:57:57.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.771+0000 7f7c7a4b3640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:57.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.771+0000 7f7c7a4b3640 4 set_mon_vals no callback set 2024-03-29T10:57:57.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.772+0000 7f7c7a4b3640 10 monclient: shutdown 2024-03-29T10:57:57.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.772+0000 7f7c7a4b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:57.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.772+0000 7f7c7a4b3640 1 -- 172.21.15.112:0/2381534417 >> v1:172.21.15.118:6789/0 conn(0x7f7c740a9750 legacy=0x7f7c740ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:57.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.772+0000 7f7c7a4b3640 1 -- 172.21.15.112:0/2381534417 shutdown_connections 2024-03-29T10:57:57.754 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.772+0000 7f7c7a4b3640 1 -- 172.21.15.112:0/2381534417 >> 172.21.15.112:0/2381534417 conn(0x7f7c740673e0 msgr2=0x7f7c740b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:57.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.775+0000 7f7c7a4b3640 1 -- 172.21.15.112:0/2381534417 shutdown_connections 2024-03-29T10:57:57.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.778+0000 7f7c7a4b3640 1 -- 172.21.15.112:0/2381534417 wait complete. 2024-03-29T10:57:57.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.779+0000 7f7c7a4b3640 10 monclient: build_initial_monmap 2024-03-29T10:57:57.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.779+0000 7f7c7a4b3640 10 monclient: monmap: 2024-03-29T10:57:57.763 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:57:57.763 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:57.764 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:57:57.764 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:57:57.764 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:57.764 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:57.764 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:57:57.764 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:57:57.764 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:57:57.764 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:57.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.779+0000 7f7c7a4b3640 1 librados: starting msgr at 2024-03-29T10:57:57.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.779+0000 7f7c7a4b3640 1 librados: starting objecter 2024-03-29T10:57:57.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.779+0000 7f7c7a4b3640 1 Processor -- start 2024-03-29T10:57:57.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.781+0000 7f7c7a4b3640 1 -- start start 2024-03-29T10:57:57.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.781+0000 7f7c7a4b3640 1 librados: setting wanted keys 2024-03-29T10:57:57.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.781+0000 7f7c7a4b3640 1 librados: calling monclient init 2024-03-29T10:57:57.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.781+0000 7f7c7a4b3640 10 monclient: init 2024-03-29T10:57:57.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.782+0000 7f7c7a4b3640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:57.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.782+0000 7f7c7a4b3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:57:57.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.782+0000 7f7c7a4b3640 10 monclient(hunting): picked mon.noname-b con 0x7f7c740a9750 addr v1:172.21.15.118:6789/0 2024-03-29T10:57:57.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.782+0000 7f7c7a4b3640 10 monclient(hunting): picked mon.noname-a con 0x7f7c740a9b30 addr v1:172.21.15.112:6789/0 2024-03-29T10:57:57.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.782+0000 7f7c7a4b3640 10 monclient(hunting): picked mon.noname-c con 0x7f7c740aad20 addr v1:172.21.15.112:6790/0 2024-03-29T10:57:57.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.782+0000 7f7c7a4b3640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c74130530 con 0x7f7c740a9b30 2024-03-29T10:57:57.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.782+0000 7f7c7a4b3640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c7406e9b0 con 0x7f7c740a9750 2024-03-29T10:57:57.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.782+0000 7f7c7a4b3640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c7406eb40 con 0x7f7c740aad20 2024-03-29T10:57:57.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.782+0000 7f7c7a4b3640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:57.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.782+0000 7f7c7a4b3640 10 monclient(hunting): authenticate will time out at 1233.086538s 2024-03-29T10:57:57.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.786+0000 7f7c79cb2640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f7c740aad20 0x7f7c741ee630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:42265/0 (socket says 172.21.15.112:42265) 2024-03-29T10:57:57.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.786+0000 7f7c79cb2640 1 -- 172.21.15.112:0/304439716 learned_addr learned my addr 172.21.15.112:0/304439716 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:57:57.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.786+0000 7f7c69ffb640 10 client.?.objecter ms_handle_connect 0x7f7c740aad20 2024-03-29T10:57:57.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.786+0000 7f7c69ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:57.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.788+0000 7f7c69ffb640 1 -- 172.21.15.112:0/304439716 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 917390520 0 0) 0x7f7c7406eb40 con 0x7f7c740aad20 2024-03-29T10:57:57.770 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.788+0000 7f7c69ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:57.770 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.788+0000 7f7c69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:57.770 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.788+0000 7f7c69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:57.771 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.788+0000 7f7c69ffb640 10 monclient(hunting): my global_id is 4487 2024-03-29T10:57:57.771 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.788+0000 7f7c69ffb640 1 -- 172.21.15.112:0/304439716 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c58004eb0 con 0x7f7c740aad20 2024-03-29T10:57:57.771 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.790+0000 7f7c69ffb640 1 -- 172.21.15.112:0/304439716 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 933675414 0 0) 0x7f7c58004eb0 con 0x7f7c740aad20 2024-03-29T10:57:57.771 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.790+0000 7f7c69ffb640 1 -- 172.21.15.112:0/304439716 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c7406eb40 con 0x7f7c740aad20 2024-03-29T10:57:57.771 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.790+0000 7f7c69ffb640 1 -- 172.21.15.112:0/304439716 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f7c6c001b50 con 0x7f7c740aad20 2024-03-29T10:57:57.771 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.790+0000 7f7c69ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:57.771 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.790+0000 7f7c69ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:57.771 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.790+0000 7f7c69ffb640 10 monclient(hunting): dump: 2024-03-29T10:57:57.772 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:57.772 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:57.772 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:57.772 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:57.772 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:57.772 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:57.772 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:57.772 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:57.772 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:57.772 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:57.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.790+0000 7f7c69ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:57.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.792+0000 7f7c69ffb640 10 client.?.objecter ms_handle_connect 0x7f7c740a9750 2024-03-29T10:57:57.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.792+0000 7f7c69ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:57.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.792+0000 7f7c69ffb640 1 -- 172.21.15.112:0/304439716 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746333057 0 0) 0x7f7c7406e9b0 con 0x7f7c740a9750 2024-03-29T10:57:57.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.792+0000 7f7c69ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:57.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.792+0000 7f7c69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:57.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.792+0000 7f7c69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:57.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.792+0000 7f7c69ffb640 10 monclient(hunting): my global_id is 4456 2024-03-29T10:57:57.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.792+0000 7f7c69ffb640 1 -- 172.21.15.112:0/304439716 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c58004eb0 con 0x7f7c740a9750 2024-03-29T10:57:57.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.792+0000 7f7c69ffb640 1 -- 172.21.15.112:0/304439716 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3870451775 0 0) 0x7f7c7406eb40 con 0x7f7c740aad20 2024-03-29T10:57:57.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.792+0000 7f7c69ffb640 1 -- 172.21.15.112:0/304439716 >> v1:172.21.15.118:6789/0 conn(0x7f7c740a9750 legacy=0x7f7c74132f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:57.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.793+0000 7f7c69ffb640 1 -- 172.21.15.112:0/304439716 >> v1:172.21.15.112:6789/0 conn(0x7f7c740a9b30 legacy=0x7f7c74071430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:57:57.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.793+0000 7f7c69ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:57:57.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.793+0000 7f7c69ffb640 1 monclient: found mon.c 2024-03-29T10:57:57.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.793+0000 7f7c69ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:57.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.793+0000 7f7c69ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:57.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.793+0000 7f7c69ffb640 1 -- 172.21.15.112:0/304439716 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c74133650 con 0x7f7c740aad20 2024-03-29T10:57:57.775 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.793+0000 7f7c69ffb640 10 monclient: _finish_auth 0 2024-03-29T10:57:57.775 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.793+0000 7f7c69ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:57:57.775 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.793+0000 7f7c69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:57.775 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.793+0000 7f7c69ffb640 1 -- 172.21.15.112:0/304439716 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c6c001cb0 con 0x7f7c740aad20 2024-03-29T10:57:57.775 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.793+0000 7f7c69ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:57.775 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.793+0000 7f7c6affd640 4 set_mon_vals no callback set 2024-03-29T10:57:57.775 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.794+0000 7f7c7a4b3640 5 monclient: authenticate success, global_id 4487 2024-03-29T10:57:57.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.794+0000 7f7c7a4b3640 10 monclient: _renew_subs 2024-03-29T10:57:57.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.794+0000 7f7c7a4b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:57.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.794+0000 7f7c7a4b3640 1 -- 172.21.15.112:0/304439716 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c741338a0 con 0x7f7c740aad20 2024-03-29T10:57:57.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.794+0000 7f7c7a4b3640 10 client.4487.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:57.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.794+0000 7f7c7a4b3640 10 monclient: _renew_subs 2024-03-29T10:57:57.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.794+0000 7f7c7a4b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:57.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.794+0000 7f7c7a4b3640 1 -- 172.21.15.112:0/304439716 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7408b770 con 0x7f7c740aad20 2024-03-29T10:57:57.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.794+0000 7f7c7a4b3640 1 librados: init done 2024-03-29T10:57:57.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.794+0000 7f7c69ffb640 1 -- 172.21.15.112:0/304439716 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f7c6c004dc0 con 0x7f7c740aad20 2024-03-29T10:57:57.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.794+0000 7f7c69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:57.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.794+0000 7f7c69ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:57.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.794+0000 7f7c69ffb640 10 monclient: dump: 2024-03-29T10:57:57.776 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:57:57.777 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:57:57.777 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:57.777 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:57:57.777 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:57.777 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:57:57.777 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:57:57.777 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:57:57.777 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:57:57.777 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:57:57.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.798+0000 7f7c69ffb640 1 -- 172.21.15.112:0/304439716 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f7c6c01ad90 con 0x7f7c740aad20 2024-03-29T10:57:57.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.799+0000 7f7c794b1640 10 monclient: get_auth_request con 0x7f7c5806dc90 auth_method 0 2024-03-29T10:57:57.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.799+0000 7f7c69ffb640 1 -- 172.21.15.112:0/304439716 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f7c6c084dc0 con 0x7f7c740aad20 2024-03-29T10:57:57.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.799+0000 7f7c69ffb640 10 client.4487.objecter ms_dispatch 0x7f7c74000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:57.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.799+0000 7f7c69ffb640 3 client.4487.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:57.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.799+0000 7f7c69ffb640 3 client.4487.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:57.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.799+0000 7f7c69ffb640 20 client.4487.objecter dump_active .. 0 homeless 2024-03-29T10:57:57.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.799+0000 7f7c69ffb640 10 client.4487.objecter ms_handle_connect 0x7f7c5806dc90 2024-03-29T10:57:57.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.799+0000 7f7c7a4b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:57.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.799+0000 7f7c7a4b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:57.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.799+0000 7f7c7a4b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:57.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.799+0000 7f7c7a4b3640 1 -- 172.21.15.112:0/304439716 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c7404f730 con 0x7f7c740aad20 2024-03-29T10:57:57.786 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.806+0000 7f7c69ffb640 1 -- 172.21.15.112:0/304439716 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7c6c0504d0 con 0x7f7c740aad20 2024-03-29T10:57:57.786 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.806+0000 7f7c69ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:57.787 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:57.806+0000 7f7c69ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:58.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:58.116+0000 7f7c7a4b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:58.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:58.116+0000 7f7c7a4b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:58.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:58.116+0000 7f7c7a4b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:57:58.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:58.116+0000 7f7c7a4b3640 1 -- 172.21.15.112:0/304439716 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c74074e10 con 0x7f7c740aad20 2024-03-29T10:57:58.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:58.119+0000 7f7c69ffb640 1 -- 172.21.15.112:0/304439716 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f7c6c0525d0 con 0x7f7c740aad20 2024-03-29T10:57:58.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:58.119+0000 7f7c69ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:58.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:58.119+0000 7f7c69ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:58.102 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:57:58.103 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:58.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:58.123+0000 7f7c7a4b3640 10 librados: watch_flush enter 2024-03-29T10:57:58.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:58.123+0000 7f7c7a4b3640 10 librados: watch_flush exit 2024-03-29T10:57:58.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:58.123+0000 7f7c7a4b3640 20 client.4487.objecter shutdown clearing up homeless session... 2024-03-29T10:57:58.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:58.123+0000 7f7c7a4b3640 10 client.4487.objecter successfully canceled tick 2024-03-29T10:57:58.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:58.123+0000 7f7c7a4b3640 1 -- 172.21.15.112:0/304439716 >> v1:172.21.15.118:6816/3181073408 conn(0x7f7c5806dc90 legacy=0x7f7c58070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:58.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:58.123+0000 7f7c7a4b3640 10 monclient: shutdown 2024-03-29T10:57:58.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:58.123+0000 7f7c7a4b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:58.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:58.123+0000 7f7c7a4b3640 1 -- 172.21.15.112:0/304439716 >> v1:172.21.15.112:6790/0 conn(0x7f7c740aad20 legacy=0x7f7c741ee630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:58.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:58.123+0000 7f7c7a4b3640 1 -- 172.21.15.112:0/304439716 shutdown_connections 2024-03-29T10:57:58.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:58.123+0000 7f7c7a4b3640 1 -- 172.21.15.112:0/304439716 >> 172.21.15.112:0/304439716 conn(0x7f7c740673e0 msgr2=0x7f7c740ac360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:58.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:58.124+0000 7f7c7a4b3640 1 -- 172.21.15.112:0/304439716 shutdown_connections 2024-03-29T10:57:58.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:58.124+0000 7f7c7a4b3640 1 -- 172.21.15.112:0/304439716 wait complete. 2024-03-29T10:57:58.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:57:58.124+0000 7f7c7a4b3640 1 librados: shutdown 2024-03-29T10:57:58.124 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:04.931 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:05.084 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.103+0000 7f3c0b0d6640 10 monclient: get_monmap_and_config 2024-03-29T10:58:05.084 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.103+0000 7f3c0b0d6640 10 monclient: build_initial_monmap 2024-03-29T10:58:05.084 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.103+0000 7f3c0b0d6640 10 monclient: monmap: 2024-03-29T10:58:05.084 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:58:05.084 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:05.084 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:58:05.104643+0000 2024-03-29T10:58:05.084 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:58:05.104643+0000 2024-03-29T10:58:05.084 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:05.084 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:05.085 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:58:05.085 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:58:05.085 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:58:05.085 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:05.086 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.104+0000 7f3c0b0d6640 1 Processor -- start 2024-03-29T10:58:05.086 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.104+0000 7f3c0b0d6640 1 -- start start 2024-03-29T10:58:05.086 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.104+0000 7f3c0b0d6640 10 monclient: init 2024-03-29T10:58:05.086 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.105+0000 7f3c0b0d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:05.086 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.105+0000 7f3c0b0d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:58:05.087 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.105+0000 7f3c0b0d6640 10 monclient(hunting): picked mon.noname-b con 0x7f3c040b2800 addr v1:172.21.15.112:6790/0 2024-03-29T10:58:05.087 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.105+0000 7f3c0b0d6640 10 monclient(hunting): picked mon.noname-c con 0x7f3c040a9c00 addr v1:172.21.15.118:6789/0 2024-03-29T10:58:05.087 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.105+0000 7f3c0b0d6640 10 monclient(hunting): picked mon.noname-a con 0x7f3c040b38c0 addr v1:172.21.15.112:6789/0 2024-03-29T10:58:05.087 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.105+0000 7f3c0b0d6640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c04141eb0 con 0x7f3c040b38c0 2024-03-29T10:58:05.087 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.105+0000 7f3c0b0d6640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c040b22d0 con 0x7f3c040a9c00 2024-03-29T10:58:05.087 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.105+0000 7f3c0b0d6640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c04142d30 con 0x7f3c040b2800 2024-03-29T10:58:05.087 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.105+0000 7f3c0b0d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:05.087 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.105+0000 7f3c0b0d6640 10 monclient(hunting): authenticate will time out at 1240.409822s 2024-03-29T10:58:05.088 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.108+0000 7f3c0a0d4640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f3c040b2800 0x7f3c04142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:49544/0 (socket says 172.21.15.112:49544) 2024-03-29T10:58:05.088 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.108+0000 7f3c0a0d4640 1 -- 172.21.15.112:0/3374811488 learned_addr learned my addr 172.21.15.112:0/3374811488 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:58:05.089 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 1 -- 172.21.15.112:0/3374811488 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3432315899 0 0) 0x7f3c04142d30 con 0x7f3c040b2800 2024-03-29T10:58:05.089 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 10 monclient(hunting): _negotiate 2024-03-29T10:58:05.089 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:05.089 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:05.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 10 monclient(hunting): my global_id is 4490 2024-03-29T10:58:05.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 1 -- 172.21.15.112:0/3374811488 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bf0003520 con 0x7f3c040b2800 2024-03-29T10:58:05.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 1 -- 172.21.15.112:0/3374811488 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388797590 0 0) 0x7f3c040b22d0 con 0x7f3c040a9c00 2024-03-29T10:58:05.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 10 monclient(hunting): _negotiate 2024-03-29T10:58:05.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:05.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:05.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 10 monclient(hunting): my global_id is 4459 2024-03-29T10:58:05.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 1 -- 172.21.15.112:0/3374811488 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c04142d30 con 0x7f3c040a9c00 2024-03-29T10:58:05.093 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 1 -- 172.21.15.112:0/3374811488 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1974613388 0 0) 0x7f3bf0003520 con 0x7f3c040b2800 2024-03-29T10:58:05.093 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 1 -- 172.21.15.112:0/3374811488 >> v1:172.21.15.118:6789/0 conn(0x7f3c040a9c00 legacy=0x7f3c0413bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:05.094 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 1 -- 172.21.15.112:0/3374811488 >> v1:172.21.15.112:6789/0 conn(0x7f3c040b38c0 legacy=0x7f3c0414d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:05.094 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 10 monclient: _finish_hunting 0 2024-03-29T10:58:05.094 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 1 monclient: found mon.noname-b 2024-03-29T10:58:05.094 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:05.094 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:58:05.094 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 1 -- 172.21.15.112:0/3374811488 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c040aaf90 con 0x7f3c040b2800 2024-03-29T10:58:05.094 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 10 monclient: _finish_auth 0 2024-03-29T10:58:05.094 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 10 monclient: _check_auth_tickets 2024-03-29T10:58:05.094 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:05.094 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 1 -- 172.21.15.112:0/3374811488 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f3c00001690 con 0x7f3c040b2800 2024-03-29T10:58:05.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.109+0000 7f3c090d2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:05.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.110+0000 7f3c090d2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:05.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.110+0000 7f3c090d2640 10 monclient: dump: 2024-03-29T10:58:05.095 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:05.095 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:05.095 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:05.095 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:05.095 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:05.095 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:05.095 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:05.096 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:05.096 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:05.096 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:05.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.110+0000 7f3c0b0d6640 5 monclient: authenticate success, global_id 4490 2024-03-29T10:58:05.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.110+0000 7f3c0b0d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:05.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.110+0000 7f3c090d2640 1 -- 172.21.15.112:0/3374811488 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c000017f0 con 0x7f3c040b2800 2024-03-29T10:58:05.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.110+0000 7f3c090d2640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:05.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.110+0000 7f3c0b0d6640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:05.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.110+0000 7f3c0b0d6640 4 set_mon_vals no callback set 2024-03-29T10:58:05.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.111+0000 7f3c0b0d6640 10 monclient: shutdown 2024-03-29T10:58:05.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.111+0000 7f3c0b0d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:05.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.111+0000 7f3c0b0d6640 1 -- 172.21.15.112:0/3374811488 >> v1:172.21.15.112:6790/0 conn(0x7f3c040b2800 legacy=0x7f3c04142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:05.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.112+0000 7f3c0b0d6640 1 -- 172.21.15.112:0/3374811488 shutdown_connections 2024-03-29T10:58:05.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.112+0000 7f3c0b0d6640 1 -- 172.21.15.112:0/3374811488 >> 172.21.15.112:0/3374811488 conn(0x7f3c04066a00 msgr2=0x7f3c04066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:05.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.113+0000 7f3c0b0d6640 1 -- 172.21.15.112:0/3374811488 shutdown_connections 2024-03-29T10:58:05.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.113+0000 7f3c0b0d6640 1 -- 172.21.15.112:0/3374811488 wait complete. 2024-03-29T10:58:05.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.114+0000 7f3c0b0d6640 10 monclient: build_initial_monmap 2024-03-29T10:58:05.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.114+0000 7f3c0b0d6640 10 monclient: monmap: 2024-03-29T10:58:05.099 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:58:05.099 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:05.099 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:58:05.099 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:58:05.099 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:05.099 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:05.099 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:58:05.099 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:58:05.099 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:58:05.099 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:05.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.115+0000 7f3c0b0d6640 1 librados: starting msgr at 2024-03-29T10:58:05.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.115+0000 7f3c0b0d6640 1 librados: starting objecter 2024-03-29T10:58:05.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.115+0000 7f3c0b0d6640 1 Processor -- start 2024-03-29T10:58:05.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.115+0000 7f3c0b0d6640 1 -- start start 2024-03-29T10:58:05.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.115+0000 7f3c0b0d6640 1 librados: setting wanted keys 2024-03-29T10:58:05.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.115+0000 7f3c0b0d6640 1 librados: calling monclient init 2024-03-29T10:58:05.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.115+0000 7f3c0b0d6640 10 monclient: init 2024-03-29T10:58:05.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.116+0000 7f3c0b0d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:05.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.116+0000 7f3c0b0d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:58:05.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.116+0000 7f3c0b0d6640 10 monclient(hunting): picked mon.noname-a con 0x7f3c040a9c00 addr v1:172.21.15.112:6789/0 2024-03-29T10:58:05.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.116+0000 7f3c0b0d6640 10 monclient(hunting): picked mon.noname-c con 0x7f3c040b2800 addr v1:172.21.15.112:6790/0 2024-03-29T10:58:05.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.116+0000 7f3c0b0d6640 10 monclient(hunting): picked mon.noname-b con 0x7f3c040b38c0 addr v1:172.21.15.118:6789/0 2024-03-29T10:58:05.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.116+0000 7f3c0b0d6640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c041d70b0 con 0x7f3c040a9c00 2024-03-29T10:58:05.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.116+0000 7f3c0b0d6640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c041d7390 con 0x7f3c040b38c0 2024-03-29T10:58:05.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.116+0000 7f3c0b0d6640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c041d7f30 con 0x7f3c040b2800 2024-03-29T10:58:05.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.116+0000 7f3c0b0d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:05.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.116+0000 7f3c0b0d6640 10 monclient(hunting): authenticate will time out at 1240.420430s 2024-03-29T10:58:05.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.118+0000 7f3c098d3640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f3c040b2800 0x7f3c040b0ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:51034/0 (socket says 172.21.15.112:51034) 2024-03-29T10:58:05.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.118+0000 7f3c098d3640 1 -- 172.21.15.112:0/438317693 learned_addr learned my addr 172.21.15.112:0/438317693 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:58:05.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.119+0000 7f3bfaffd640 10 client.?.objecter ms_handle_connect 0x7f3c040b2800 2024-03-29T10:58:05.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.119+0000 7f3bfaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:05.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.119+0000 7f3bfaffd640 1 -- 172.21.15.112:0/438317693 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342688559 0 0) 0x7f3c041d7f30 con 0x7f3c040b2800 2024-03-29T10:58:05.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.119+0000 7f3bfaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:05.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.119+0000 7f3bfaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:05.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.119+0000 7f3bfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:05.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.119+0000 7f3bfaffd640 10 monclient(hunting): my global_id is 4493 2024-03-29T10:58:05.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.119+0000 7f3bfaffd640 1 -- 172.21.15.112:0/438317693 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be8004eb0 con 0x7f3c040b2800 2024-03-29T10:58:05.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.119+0000 7f3bfaffd640 10 client.?.objecter ms_handle_connect 0x7f3c040a9c00 2024-03-29T10:58:05.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.119+0000 7f3bfaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:05.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.120+0000 7f3bfaffd640 1 -- 172.21.15.112:0/438317693 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911770380 0 0) 0x7f3c041d70b0 con 0x7f3c040a9c00 2024-03-29T10:58:05.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.120+0000 7f3bfaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:05.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.120+0000 7f3bfaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:05.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.120+0000 7f3bfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:05.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.120+0000 7f3bfaffd640 10 monclient(hunting): my global_id is 4464 2024-03-29T10:58:05.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.120+0000 7f3bfaffd640 1 -- 172.21.15.112:0/438317693 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c041d7f30 con 0x7f3c040a9c00 2024-03-29T10:58:05.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.120+0000 7f3bfaffd640 1 -- 172.21.15.112:0/438317693 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1741327158 0 0) 0x7f3be8004eb0 con 0x7f3c040b2800 2024-03-29T10:58:05.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.120+0000 7f3bfaffd640 1 -- 172.21.15.112:0/438317693 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c041d70b0 con 0x7f3c040b2800 2024-03-29T10:58:05.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.120+0000 7f3bfaffd640 1 -- 172.21.15.112:0/438317693 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f3bf4001970 con 0x7f3c040b2800 2024-03-29T10:58:05.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.120+0000 7f3bfaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:05.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.120+0000 7f3bfaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:05.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.120+0000 7f3bfaffd640 10 monclient(hunting): dump: 2024-03-29T10:58:05.105 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:05.105 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:05.105 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:05.105 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:05.105 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:05.105 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:05.105 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:05.105 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:05.106 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:05.106 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:05.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.120+0000 7f3bfaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:05.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.120+0000 7f3bfaffd640 1 -- 172.21.15.112:0/438317693 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3172409577 0 0) 0x7f3c041d70b0 con 0x7f3c040b2800 2024-03-29T10:58:05.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.120+0000 7f3bfaffd640 1 -- 172.21.15.112:0/438317693 >> v1:172.21.15.118:6789/0 conn(0x7f3c040b38c0 legacy=0x7f3c041e29b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:58:05.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.122+0000 7f3bfaffd640 1 -- 172.21.15.112:0/438317693 >> v1:172.21.15.112:6789/0 conn(0x7f3c040a9c00 legacy=0x7f3c041d7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:05.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.122+0000 7f3bfaffd640 10 monclient: _finish_hunting 0 2024-03-29T10:58:05.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.122+0000 7f3bfaffd640 1 monclient: found mon.c 2024-03-29T10:58:05.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.122+0000 7f3bfaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:05.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.122+0000 7f3bfaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:58:05.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.122+0000 7f3bfaffd640 1 -- 172.21.15.112:0/438317693 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c040ad070 con 0x7f3c040b2800 2024-03-29T10:58:05.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.122+0000 7f3bfaffd640 10 monclient: _finish_auth 0 2024-03-29T10:58:05.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.122+0000 7f3bfaffd640 10 monclient: _check_auth_tickets 2024-03-29T10:58:05.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.122+0000 7f3bfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:05.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.122+0000 7f3c0b0d6640 5 monclient: authenticate success, global_id 4493 2024-03-29T10:58:05.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.122+0000 7f3c0b0d6640 10 monclient: _renew_subs 2024-03-29T10:58:05.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.122+0000 7f3c0b0d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:58:05.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.122+0000 7f3c0b0d6640 1 -- 172.21.15.112:0/438317693 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c040abd00 con 0x7f3c040b2800 2024-03-29T10:58:05.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.122+0000 7f3c0b0d6640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:05.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.122+0000 7f3c0b0d6640 10 monclient: _renew_subs 2024-03-29T10:58:05.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.122+0000 7f3c0b0d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:58:05.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.122+0000 7f3c0b0d6640 1 -- 172.21.15.112:0/438317693 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c0413c720 con 0x7f3c040b2800 2024-03-29T10:58:05.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.122+0000 7f3c0b0d6640 1 librados: init done 2024-03-29T10:58:05.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.123+0000 7f3bfaffd640 1 -- 172.21.15.112:0/438317693 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3bf40035f0 con 0x7f3c040b2800 2024-03-29T10:58:05.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.123+0000 7f3bfaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:05.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.123+0000 7f3bfaffd640 1 -- 172.21.15.112:0/438317693 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f3bf4003a00 con 0x7f3c040b2800 2024-03-29T10:58:05.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.123+0000 7f3bfaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:05.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.123+0000 7f3bfaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:05.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.123+0000 7f3bfaffd640 10 monclient: dump: 2024-03-29T10:58:05.108 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:05.108 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:05.108 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:05.108 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:05.108 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:05.108 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:05.108 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:05.108 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:05.108 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:05.108 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:05.109 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.125+0000 7f3bfaffd640 1 -- 172.21.15.112:0/438317693 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f3bf4003b60 con 0x7f3c040b2800 2024-03-29T10:58:05.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.126+0000 7f3bfaffd640 1 -- 172.21.15.112:0/438317693 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f3bf4084a50 con 0x7f3c040b2800 2024-03-29T10:58:05.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.126+0000 7f3bfaffd640 10 client.4493.objecter ms_dispatch 0x7f3c04000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:05.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.126+0000 7f3bfaffd640 3 client.4493.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:05.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.126+0000 7f3bfaffd640 3 client.4493.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:05.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.126+0000 7f3bfaffd640 20 client.4493.objecter dump_active .. 0 homeless 2024-03-29T10:58:05.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.125+0000 7f3c088d1640 4 set_mon_vals no callback set 2024-03-29T10:58:05.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.127+0000 7f3c0a0d4640 10 monclient: get_auth_request con 0x7f3be806dd00 auth_method 0 2024-03-29T10:58:05.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.126+0000 7f3c0b0d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:05.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.127+0000 7f3c0b0d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:05.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.127+0000 7f3c0b0d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:58:05.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.127+0000 7f3c0b0d6640 1 -- 172.21.15.112:0/438317693 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bec005080 con 0x7f3c040b2800 2024-03-29T10:58:05.111 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.130+0000 7f3bfaffd640 10 client.4493.objecter ms_handle_connect 0x7f3be806dd00 2024-03-29T10:58:05.117 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.136+0000 7f3bfaffd640 1 -- 172.21.15.112:0/438317693 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3bf4050330 con 0x7f3c040b2800 2024-03-29T10:58:05.117 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.136+0000 7f3bfaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:05.117 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.136+0000 7f3bfaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:05.441 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.461+0000 7f3c0b0d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:05.441 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.461+0000 7f3c0b0d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:05.441 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.461+0000 7f3c0b0d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:58:05.441 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.461+0000 7f3c0b0d6640 1 -- 172.21.15.112:0/438317693 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3bec005330 con 0x7f3c040b2800 2024-03-29T10:58:05.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.463+0000 7f3bfaffd640 1 -- 172.21.15.112:0/438317693 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f3bf4052260 con 0x7f3c040b2800 2024-03-29T10:58:05.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.463+0000 7f3bfaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:05.444 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.463+0000 7f3bfaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:05.444 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:58:05.445 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:05.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.467+0000 7f3c0b0d6640 10 librados: watch_flush enter 2024-03-29T10:58:05.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.467+0000 7f3c0b0d6640 10 librados: watch_flush exit 2024-03-29T10:58:05.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.467+0000 7f3c0b0d6640 20 client.4493.objecter shutdown clearing up homeless session... 2024-03-29T10:58:05.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.467+0000 7f3c0b0d6640 10 client.4493.objecter successfully canceled tick 2024-03-29T10:58:05.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.467+0000 7f3c0b0d6640 1 -- 172.21.15.112:0/438317693 >> v1:172.21.15.118:6816/3181073408 conn(0x7f3be806dd00 legacy=0x7f3be80701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:05.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.467+0000 7f3c0b0d6640 10 monclient: shutdown 2024-03-29T10:58:05.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.467+0000 7f3c0b0d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:05.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.467+0000 7f3c0b0d6640 1 -- 172.21.15.112:0/438317693 >> v1:172.21.15.112:6790/0 conn(0x7f3c040b2800 legacy=0x7f3c040b0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:05.449 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.467+0000 7f3c0b0d6640 1 -- 172.21.15.112:0/438317693 shutdown_connections 2024-03-29T10:58:05.449 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.467+0000 7f3c0b0d6640 1 -- 172.21.15.112:0/438317693 >> 172.21.15.112:0/438317693 conn(0x7f3c04066a00 msgr2=0x7f3c0413bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:05.449 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.467+0000 7f3c0b0d6640 1 -- 172.21.15.112:0/438317693 shutdown_connections 2024-03-29T10:58:05.449 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.467+0000 7f3c0b0d6640 1 -- 172.21.15.112:0/438317693 wait complete. 2024-03-29T10:58:05.449 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:05.468+0000 7f3c0b0d6640 1 librados: shutdown 2024-03-29T10:58:05.465 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:12.272 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:12.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.446+0000 7f23bf73b640 10 monclient: get_monmap_and_config 2024-03-29T10:58:12.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.446+0000 7f23bf73b640 10 monclient: build_initial_monmap 2024-03-29T10:58:12.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.446+0000 7f23bf73b640 10 monclient: monmap: 2024-03-29T10:58:12.429 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:58:12.429 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:12.429 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:58:12.447241+0000 2024-03-29T10:58:12.429 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:58:12.447241+0000 2024-03-29T10:58:12.429 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:12.430 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:12.430 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:58:12.430 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:58:12.430 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:58:12.430 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:12.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.448+0000 7f23bf73b640 1 Processor -- start 2024-03-29T10:58:12.431 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.449+0000 7f23bf73b640 1 -- start start 2024-03-29T10:58:12.431 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.449+0000 7f23bf73b640 10 monclient: init 2024-03-29T10:58:12.431 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.449+0000 7f23bf73b640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:12.431 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.449+0000 7f23bf73b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:58:12.431 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.449+0000 7f23bf73b640 10 monclient(hunting): picked mon.noname-a con 0x7f23b80a9ac0 addr v1:172.21.15.112:6789/0 2024-03-29T10:58:12.431 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.449+0000 7f23bf73b640 10 monclient(hunting): picked mon.noname-c con 0x7f23b813ee90 addr v1:172.21.15.118:6789/0 2024-03-29T10:58:12.431 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.449+0000 7f23bf73b640 10 monclient(hunting): picked mon.noname-b con 0x7f23b80af900 addr v1:172.21.15.112:6790/0 2024-03-29T10:58:12.431 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.449+0000 7f23bf73b640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b80afd10 con 0x7f23b80a9ac0 2024-03-29T10:58:12.431 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.449+0000 7f23bf73b640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b80af6f0 con 0x7f23b813ee90 2024-03-29T10:58:12.431 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.449+0000 7f23bf73b640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b80aac80 con 0x7f23b80af900 2024-03-29T10:58:12.431 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.449+0000 7f23bf73b640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:12.431 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.449+0000 7f23bf73b640 10 monclient(hunting): authenticate will time out at 1247.754170s 2024-03-29T10:58:12.433 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.451+0000 7f23bdcd8640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f23b80af900 0x7f23b8149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:32824/0 (socket says 172.21.15.112:32824) 2024-03-29T10:58:12.433 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.451+0000 7f23bdcd8640 1 -- 172.21.15.112:0/3683447209 learned_addr learned my addr 172.21.15.112:0/3683447209 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:58:12.433 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.452+0000 7f23affff640 1 -- 172.21.15.112:0/3683447209 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099203716 0 0) 0x7f23b80aac80 con 0x7f23b80af900 2024-03-29T10:58:12.434 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.452+0000 7f23affff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:12.434 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.452+0000 7f23affff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:12.434 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.452+0000 7f23affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:12.434 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.452+0000 7f23affff640 10 monclient(hunting): my global_id is 4496 2024-03-29T10:58:12.434 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.452+0000 7f23affff640 1 -- 172.21.15.112:0/3683447209 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23a4003500 con 0x7f23b80af900 2024-03-29T10:58:12.434 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.452+0000 7f23affff640 1 -- 172.21.15.112:0/3683447209 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4103517517 0 0) 0x7f23a4003500 con 0x7f23b80af900 2024-03-29T10:58:12.434 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.452+0000 7f23affff640 1 -- 172.21.15.112:0/3683447209 >> v1:172.21.15.118:6789/0 conn(0x7f23b813ee90 legacy=0x7f23b813b5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:58:12.434 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.452+0000 7f23affff640 1 -- 172.21.15.112:0/3683447209 >> v1:172.21.15.112:6789/0 conn(0x7f23b80a9ac0 legacy=0x7f23b80adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:58:12.434 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.453+0000 7f23affff640 10 monclient: _finish_hunting 0 2024-03-29T10:58:12.434 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.453+0000 7f23affff640 1 monclient: found mon.noname-b 2024-03-29T10:58:12.434 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.453+0000 7f23affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:12.434 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.453+0000 7f23affff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:58:12.434 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.453+0000 7f23affff640 1 -- 172.21.15.112:0/3683447209 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b80aaf60 con 0x7f23b80af900 2024-03-29T10:58:12.435 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.453+0000 7f23affff640 10 monclient: _finish_auth 0 2024-03-29T10:58:12.435 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.453+0000 7f23affff640 10 monclient: _check_auth_tickets 2024-03-29T10:58:12.435 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.453+0000 7f23affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:12.435 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.453+0000 7f23affff640 1 -- 172.21.15.112:0/3683447209 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f23a8004cb0 con 0x7f23b80af900 2024-03-29T10:58:12.435 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.453+0000 7f23affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:12.435 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.453+0000 7f23affff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:12.435 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.453+0000 7f23affff640 10 monclient: dump: 2024-03-29T10:58:12.435 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:12.435 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:12.435 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:12.435 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:12.435 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:12.435 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:12.436 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:12.436 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:12.436 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:12.436 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:12.436 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.453+0000 7f23affff640 1 -- 172.21.15.112:0/3683447209 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23a8004f10 con 0x7f23b80af900 2024-03-29T10:58:12.436 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.453+0000 7f23affff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:12.436 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.453+0000 7f23affff640 1 -- 172.21.15.112:0/3683447209 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f23a80014b0 con 0x7f23b80af900 2024-03-29T10:58:12.436 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.453+0000 7f23affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:12.436 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.453+0000 7f23affff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:12.436 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.453+0000 7f23affff640 10 monclient: dump: 2024-03-29T10:58:12.436 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:12.436 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:12.436 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:12.436 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:12.436 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:12.437 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:12.437 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:12.437 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:12.437 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:12.437 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:12.437 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.454+0000 7f23bf73b640 5 monclient: authenticate success, global_id 4496 2024-03-29T10:58:12.438 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.454+0000 7f23bf73b640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:12.438 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.454+0000 7f23bf73b640 4 set_mon_vals no callback set 2024-03-29T10:58:12.438 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.455+0000 7f23bf73b640 10 monclient: shutdown 2024-03-29T10:58:12.438 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.455+0000 7f23bf73b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:12.438 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.455+0000 7f23bf73b640 1 -- 172.21.15.112:0/3683447209 >> v1:172.21.15.112:6790/0 conn(0x7f23b80af900 legacy=0x7f23b8149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:12.438 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.455+0000 7f23bf73b640 1 -- 172.21.15.112:0/3683447209 shutdown_connections 2024-03-29T10:58:12.438 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.455+0000 7f23bf73b640 1 -- 172.21.15.112:0/3683447209 >> 172.21.15.112:0/3683447209 conn(0x7f23b80b35e0 msgr2=0x7f23b80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:12.439 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.459+0000 7f23bf73b640 1 -- 172.21.15.112:0/3683447209 shutdown_connections 2024-03-29T10:58:12.439 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.459+0000 7f23bf73b640 1 -- 172.21.15.112:0/3683447209 wait complete. 2024-03-29T10:58:12.440 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.460+0000 7f23bf73b640 10 monclient: build_initial_monmap 2024-03-29T10:58:12.441 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.460+0000 7f23bf73b640 10 monclient: monmap: 2024-03-29T10:58:12.441 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:58:12.441 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:12.441 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:58:12.441 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:58:12.441 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:12.441 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:12.441 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:58:12.441 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:58:12.441 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:58:12.442 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:12.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.460+0000 7f23bf73b640 1 librados: starting msgr at 2024-03-29T10:58:12.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.460+0000 7f23bf73b640 1 librados: starting objecter 2024-03-29T10:58:12.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.460+0000 7f23bf73b640 1 Processor -- start 2024-03-29T10:58:12.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.460+0000 7f23bf73b640 1 -- start start 2024-03-29T10:58:12.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.460+0000 7f23bf73b640 1 librados: setting wanted keys 2024-03-29T10:58:12.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.460+0000 7f23bf73b640 1 librados: calling monclient init 2024-03-29T10:58:12.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.460+0000 7f23bf73b640 10 monclient: init 2024-03-29T10:58:12.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.461+0000 7f23bf73b640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:12.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.461+0000 7f23bf73b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:58:12.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.461+0000 7f23bf73b640 10 monclient(hunting): picked mon.noname-c con 0x7f23b80a9ac0 addr v1:172.21.15.112:6790/0 2024-03-29T10:58:12.444 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.461+0000 7f23bd4d7640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f23b80a9ac0 0x7f23b80b2770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:33656/0 (socket says 172.21.15.112:33656) 2024-03-29T10:58:12.444 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.461+0000 7f23bf73b640 10 monclient(hunting): picked mon.noname-b con 0x7f23b80af900 addr v1:172.21.15.118:6789/0 2024-03-29T10:58:12.444 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.461+0000 7f23bd4d7640 1 -- 172.21.15.112:0/1786825561 learned_addr learned my addr 172.21.15.112:0/1786825561 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:58:12.444 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.462+0000 7f23adffb640 10 client.?.objecter ms_handle_connect 0x7f23b80a9ac0 2024-03-29T10:58:12.444 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.462+0000 7f23adffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:12.444 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.462+0000 7f23bf73b640 10 monclient(hunting): picked mon.noname-a con 0x7f23b813ee90 addr v1:172.21.15.112:6789/0 2024-03-29T10:58:12.444 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.462+0000 7f23bf73b640 1 -- 172.21.15.112:0/1786825561 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b80b2e80 con 0x7f23b813ee90 2024-03-29T10:58:12.444 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.462+0000 7f23bf73b640 1 -- 172.21.15.112:0/1786825561 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b8140000 con 0x7f23b80af900 2024-03-29T10:58:12.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.464+0000 7f23bf73b640 1 -- 172.21.15.112:0/1786825561 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b81402e0 con 0x7f23b80a9ac0 2024-03-29T10:58:12.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.464+0000 7f23bf73b640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:12.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.464+0000 7f23bf73b640 10 monclient(hunting): authenticate will time out at 1247.768876s 2024-03-29T10:58:12.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.464+0000 7f23adffb640 1 -- 172.21.15.112:0/1786825561 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173666291 0 0) 0x7f23b81402e0 con 0x7f23b80a9ac0 2024-03-29T10:58:12.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.464+0000 7f23adffb640 10 monclient(hunting): _negotiate 2024-03-29T10:58:12.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.464+0000 7f23adffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:12.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.464+0000 7f23adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:12.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.464+0000 7f23adffb640 10 monclient(hunting): my global_id is 4499 2024-03-29T10:58:12.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.464+0000 7f23adffb640 1 -- 172.21.15.112:0/1786825561 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f239c004eb0 con 0x7f23b80a9ac0 2024-03-29T10:58:12.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.465+0000 7f23adffb640 10 client.?.objecter ms_handle_connect 0x7f23b80af900 2024-03-29T10:58:12.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.465+0000 7f23adffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:12.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.465+0000 7f23adffb640 1 -- 172.21.15.112:0/1786825561 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1256540951 0 0) 0x7f239c004eb0 con 0x7f23b80a9ac0 2024-03-29T10:58:12.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.465+0000 7f23adffb640 1 -- 172.21.15.112:0/1786825561 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23b81402e0 con 0x7f23b80a9ac0 2024-03-29T10:58:12.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.465+0000 7f23adffb640 1 -- 172.21.15.112:0/1786825561 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f23b0001b50 con 0x7f23b80a9ac0 2024-03-29T10:58:12.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.465+0000 7f23adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:12.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.465+0000 7f23adffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:12.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.465+0000 7f23adffb640 10 monclient(hunting): dump: 2024-03-29T10:58:12.448 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:12.448 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:12.448 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:12.448 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:12.448 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:12.448 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:12.448 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:12.448 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:12.448 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:12.448 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:12.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.465+0000 7f23adffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:12.449 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.465+0000 7f23adffb640 10 client.?.objecter ms_handle_connect 0x7f23b813ee90 2024-03-29T10:58:12.449 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.465+0000 7f23adffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:12.451 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23adffb640 1 -- 172.21.15.112:0/1786825561 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1486449971 0 0) 0x7f23b80b2e80 con 0x7f23b813ee90 2024-03-29T10:58:12.451 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23adffb640 10 monclient(hunting): _negotiate 2024-03-29T10:58:12.451 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23adffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:12.451 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:12.452 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23adffb640 10 monclient(hunting): my global_id is 4467 2024-03-29T10:58:12.452 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23adffb640 1 -- 172.21.15.112:0/1786825561 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f239c004eb0 con 0x7f23b813ee90 2024-03-29T10:58:12.452 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23adffb640 1 -- 172.21.15.112:0/1786825561 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 154414094 0 0) 0x7f23b81402e0 con 0x7f23b80a9ac0 2024-03-29T10:58:12.452 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23adffb640 1 -- 172.21.15.112:0/1786825561 >> v1:172.21.15.118:6789/0 conn(0x7f23b80af900 legacy=0x7f23b813c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:12.452 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23adffb640 1 -- 172.21.15.112:0/1786825561 >> v1:172.21.15.112:6789/0 conn(0x7f23b813ee90 legacy=0x7f23b813cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:12.452 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23adffb640 10 monclient: _finish_hunting 0 2024-03-29T10:58:12.452 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23adffb640 1 monclient: found mon.c 2024-03-29T10:58:12.452 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:12.452 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23adffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:58:12.452 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23adffb640 1 -- 172.21.15.112:0/1786825561 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b80b3140 con 0x7f23b80a9ac0 2024-03-29T10:58:12.452 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23adffb640 10 monclient: _finish_auth 0 2024-03-29T10:58:12.452 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23adffb640 10 monclient: _check_auth_tickets 2024-03-29T10:58:12.452 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:12.452 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23bf73b640 5 monclient: authenticate success, global_id 4499 2024-03-29T10:58:12.453 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23bf73b640 10 monclient: _renew_subs 2024-03-29T10:58:12.453 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23bf73b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:58:12.453 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23bf73b640 1 -- 172.21.15.112:0/1786825561 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b80b1bf0 con 0x7f23b80a9ac0 2024-03-29T10:58:12.453 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23bf73b640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:12.455 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23bf73b640 10 monclient: _renew_subs 2024-03-29T10:58:12.456 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23bf73b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:58:12.456 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23bf73b640 1 -- 172.21.15.112:0/1786825561 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b81e74b0 con 0x7f23b80a9ac0 2024-03-29T10:58:12.456 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.469+0000 7f23bf73b640 1 librados: init done 2024-03-29T10:58:12.456 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.471+0000 7f23adffb640 1 -- 172.21.15.112:0/1786825561 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23b00020e0 con 0x7f23b80a9ac0 2024-03-29T10:58:12.456 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.471+0000 7f23adffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:12.456 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.471+0000 7f23adffb640 1 -- 172.21.15.112:0/1786825561 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f23b0004d80 con 0x7f23b80a9ac0 2024-03-29T10:58:12.456 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.471+0000 7f23adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:12.456 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.471+0000 7f23adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:12.456 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.471+0000 7f23adffb640 10 monclient: dump: 2024-03-29T10:58:12.456 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:12.456 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:12.456 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:12.456 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:12.457 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:12.457 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:12.457 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:12.457 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:12.457 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:12.457 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:12.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.471+0000 7f23adffb640 1 -- 172.21.15.112:0/1786825561 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f23b001ad50 con 0x7f23b80a9ac0 2024-03-29T10:58:12.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.472+0000 7f23af7fe640 4 set_mon_vals no callback set 2024-03-29T10:58:12.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.473+0000 7f23adffb640 1 -- 172.21.15.112:0/1786825561 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f23b0054c90 con 0x7f23b80a9ac0 2024-03-29T10:58:12.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.473+0000 7f23adffb640 10 client.4499.objecter ms_dispatch 0x7f23b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:12.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.473+0000 7f23adffb640 3 client.4499.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:12.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.473+0000 7f23adffb640 3 client.4499.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:12.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.473+0000 7f23adffb640 20 client.4499.objecter dump_active .. 0 homeless 2024-03-29T10:58:12.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.473+0000 7f23bccd6640 10 monclient: get_auth_request con 0x7f239c06dc90 auth_method 0 2024-03-29T10:58:12.457 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.473+0000 7f23a37fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:12.458 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.473+0000 7f23a37fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:12.458 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.473+0000 7f23a37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:58:12.458 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.473+0000 7f23a37fe640 1 -- 172.21.15.112:0/1786825561 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2398005080 con 0x7f23b80a9ac0 2024-03-29T10:58:12.458 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.474+0000 7f23adffb640 10 client.4499.objecter ms_handle_connect 0x7f239c06dc90 2024-03-29T10:58:12.466 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.486+0000 7f23adffb640 1 -- 172.21.15.112:0/1786825561 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23b00507a0 con 0x7f23b80a9ac0 2024-03-29T10:58:12.466 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.486+0000 7f23adffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:12.467 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.486+0000 7f23adffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:12.795 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.815+0000 7f23a37fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:12.795 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.815+0000 7f23a37fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:12.795 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.815+0000 7f23a37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:58:12.795 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.815+0000 7f23a37fe640 1 -- 172.21.15.112:0/1786825561 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2398005330 con 0x7f23b80a9ac0 2024-03-29T10:58:12.796 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.815+0000 7f23adffb640 1 -- 172.21.15.112:0/1786825561 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f23b00526d0 con 0x7f23b80a9ac0 2024-03-29T10:58:12.796 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.815+0000 7f23adffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:12.796 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.815+0000 7f23adffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:12.797 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:58:12.797 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:12.799 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.819+0000 7f23a37fe640 10 librados: watch_flush enter 2024-03-29T10:58:12.800 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.819+0000 7f23a37fe640 10 librados: watch_flush exit 2024-03-29T10:58:12.800 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.820+0000 7f23a37fe640 20 client.4499.objecter shutdown clearing up homeless session... 2024-03-29T10:58:12.800 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.820+0000 7f23a37fe640 10 client.4499.objecter successfully canceled tick 2024-03-29T10:58:12.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.820+0000 7f23a37fe640 1 -- 172.21.15.112:0/1786825561 >> v1:172.21.15.118:6816/3181073408 conn(0x7f239c06dc90 legacy=0x7f239c070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:12.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.821+0000 7f23a37fe640 10 monclient: shutdown 2024-03-29T10:58:12.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.821+0000 7f23a37fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:12.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.821+0000 7f23a37fe640 1 -- 172.21.15.112:0/1786825561 >> v1:172.21.15.112:6790/0 conn(0x7f23b80a9ac0 legacy=0x7f23b80b2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:12.803 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.823+0000 7f23a37fe640 1 -- 172.21.15.112:0/1786825561 shutdown_connections 2024-03-29T10:58:12.803 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.823+0000 7f23a37fe640 1 -- 172.21.15.112:0/1786825561 >> 172.21.15.112:0/1786825561 conn(0x7f23b80b35e0 msgr2=0x7f23b80adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:12.806 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.826+0000 7f23a37fe640 1 -- 172.21.15.112:0/1786825561 shutdown_connections 2024-03-29T10:58:12.806 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.826+0000 7f23a37fe640 1 -- 172.21.15.112:0/1786825561 wait complete. 2024-03-29T10:58:12.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:12.827+0000 7f23a37fe640 1 librados: shutdown 2024-03-29T10:58:12.826 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:19.634 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:19.788 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.807+0000 7fe8d8e25640 10 monclient: get_monmap_and_config 2024-03-29T10:58:19.788 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.807+0000 7fe8d8e25640 10 monclient: build_initial_monmap 2024-03-29T10:58:19.788 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.807+0000 7fe8d8e25640 10 monclient: monmap: 2024-03-29T10:58:19.788 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:58:19.788 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:19.788 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:58:19.808628+0000 2024-03-29T10:58:19.788 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:58:19.808628+0000 2024-03-29T10:58:19.788 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:19.788 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:19.788 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:58:19.788 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:58:19.789 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:58:19.789 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:19.789 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.809+0000 7fe8d8e25640 1 Processor -- start 2024-03-29T10:58:19.793 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.812+0000 7fe8d8e25640 1 -- start start 2024-03-29T10:58:19.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.812+0000 7fe8d8e25640 10 monclient: init 2024-03-29T10:58:19.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.813+0000 7fe8d8e25640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:19.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.813+0000 7fe8d8e25640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:58:19.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.813+0000 7fe8d8e25640 10 monclient(hunting): picked mon.noname-b con 0x7fe8d40b2520 addr v1:172.21.15.112:6790/0 2024-03-29T10:58:19.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.814+0000 7fe8d8e25640 10 monclient(hunting): picked mon.noname-a con 0x7fe8d40b2900 addr v1:172.21.15.112:6789/0 2024-03-29T10:58:19.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.814+0000 7fe8d8e25640 10 monclient(hunting): picked mon.noname-c con 0x7fe8d40a9c70 addr v1:172.21.15.118:6789/0 2024-03-29T10:58:19.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.814+0000 7fe8d8e25640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d4142170 con 0x7fe8d40b2900 2024-03-29T10:58:19.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.814+0000 7fe8d8e25640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d40b2260 con 0x7fe8d40a9c70 2024-03-29T10:58:19.799 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.814+0000 7fe8d8e25640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d40aaf60 con 0x7fe8d40b2520 2024-03-29T10:58:19.799 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.814+0000 7fe8d8e25640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:19.799 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.814+0000 7fe8d8e25640 10 monclient(hunting): authenticate will time out at 1255.118480s 2024-03-29T10:58:19.800 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.819+0000 7fe8d2d9d640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fe8d40a9c70 0x7fe8d414d7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:44793/0 (socket says 172.21.15.112:44793) 2024-03-29T10:58:19.800 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.820+0000 7fe8d2d9d640 1 -- 172.21.15.112:0/4188992456 learned_addr learned my addr 172.21.15.112:0/4188992456 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:58:19.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.821+0000 7fe8d159a640 1 -- 172.21.15.112:0/4188992456 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822557479 0 0) 0x7fe8d4142170 con 0x7fe8d40b2900 2024-03-29T10:58:19.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.821+0000 7fe8d159a640 10 monclient(hunting): _negotiate 2024-03-29T10:58:19.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.822+0000 7fe8d159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:19.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.822+0000 7fe8d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:19.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.822+0000 7fe8d159a640 10 monclient(hunting): my global_id is 4470 2024-03-29T10:58:19.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.822+0000 7fe8d159a640 1 -- 172.21.15.112:0/4188992456 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8bc003520 con 0x7fe8d40b2900 2024-03-29T10:58:19.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.822+0000 7fe8d159a640 1 -- 172.21.15.112:0/4188992456 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 580067951 0 0) 0x7fe8bc003520 con 0x7fe8d40b2900 2024-03-29T10:58:19.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.822+0000 7fe8d159a640 1 -- 172.21.15.112:0/4188992456 >> v1:172.21.15.112:6790/0 conn(0x7fe8d40b2520 legacy=0x7fe8d40b1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:19.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.822+0000 7fe8d159a640 1 -- 172.21.15.112:0/4188992456 >> v1:172.21.15.118:6789/0 conn(0x7fe8d40a9c70 legacy=0x7fe8d414d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:19.805 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.822+0000 7fe8d159a640 10 monclient: _finish_hunting 0 2024-03-29T10:58:19.805 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.823+0000 7fe8d159a640 1 monclient: found mon.noname-a 2024-03-29T10:58:19.805 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.823+0000 7fe8d159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:19.805 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.823+0000 7fe8d159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T10:58:19.805 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.823+0000 7fe8d159a640 1 -- 172.21.15.112:0/4188992456 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d4142d10 con 0x7fe8d40b2900 2024-03-29T10:58:19.805 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.823+0000 7fe8d159a640 10 monclient: _finish_auth 0 2024-03-29T10:58:19.805 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.823+0000 7fe8d159a640 10 monclient: _check_auth_tickets 2024-03-29T10:58:19.805 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.823+0000 7fe8d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:19.805 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.823+0000 7fe8d159a640 1 -- 172.21.15.112:0/4188992456 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe8c40015c0 con 0x7fe8d40b2900 2024-03-29T10:58:19.805 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.823+0000 7fe8d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:19.806 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.823+0000 7fe8d159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:58:19.806 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.823+0000 7fe8d159a640 10 monclient: dump: 2024-03-29T10:58:19.806 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:19.806 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:19.806 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:19.806 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:19.806 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:19.806 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:19.806 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:19.806 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:19.806 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:19.806 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:19.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.825+0000 7fe8d8e25640 5 monclient: authenticate success, global_id 4470 2024-03-29T10:58:19.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.825+0000 7fe8d8e25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:19.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.827+0000 7fe8d159a640 1 -- 172.21.15.112:0/4188992456 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8c40017e0 con 0x7fe8d40b2900 2024-03-29T10:58:19.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.827+0000 7fe8d159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:19.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.827+0000 7fe8d159a640 1 -- 172.21.15.112:0/4188992456 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe8c4001b70 con 0x7fe8d40b2900 2024-03-29T10:58:19.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.827+0000 7fe8d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:19.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.827+0000 7fe8d159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:19.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.827+0000 7fe8d159a640 10 monclient: dump: 2024-03-29T10:58:19.808 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:19.808 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:19.808 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:19.808 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:19.809 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:19.809 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:19.809 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:19.809 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:19.809 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:19.809 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:19.809 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.827+0000 7fe8d8e25640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:19.809 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.827+0000 7fe8d8e25640 4 set_mon_vals no callback set 2024-03-29T10:58:19.810 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.828+0000 7fe8d8e25640 10 monclient: shutdown 2024-03-29T10:58:19.810 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.828+0000 7fe8d8e25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:19.810 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.828+0000 7fe8d8e25640 1 -- 172.21.15.112:0/4188992456 >> v1:172.21.15.112:6789/0 conn(0x7fe8d40b2900 legacy=0x7fe8d4142600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:19.812 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.832+0000 7fe8d8e25640 1 -- 172.21.15.112:0/4188992456 shutdown_connections 2024-03-29T10:58:19.812 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.832+0000 7fe8d8e25640 1 -- 172.21.15.112:0/4188992456 >> 172.21.15.112:0/4188992456 conn(0x7fe8d406d180 msgr2=0x7fe8d4066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:19.812 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.832+0000 7fe8d8e25640 1 -- 172.21.15.112:0/4188992456 shutdown_connections 2024-03-29T10:58:19.812 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.832+0000 7fe8d8e25640 1 -- 172.21.15.112:0/4188992456 wait complete. 2024-03-29T10:58:19.815 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.833+0000 7fe8d8e25640 10 monclient: build_initial_monmap 2024-03-29T10:58:19.815 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.834+0000 7fe8d8e25640 10 monclient: monmap: 2024-03-29T10:58:19.815 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:58:19.815 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:19.815 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:58:19.815 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:58:19.815 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:19.815 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:19.815 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:58:19.815 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:58:19.815 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:58:19.815 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:19.815 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.834+0000 7fe8d8e25640 1 librados: starting msgr at 2024-03-29T10:58:19.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.834+0000 7fe8d8e25640 1 librados: starting objecter 2024-03-29T10:58:19.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.834+0000 7fe8d8e25640 1 Processor -- start 2024-03-29T10:58:19.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.834+0000 7fe8d8e25640 1 -- start start 2024-03-29T10:58:19.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.834+0000 7fe8d8e25640 1 librados: setting wanted keys 2024-03-29T10:58:19.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.834+0000 7fe8d8e25640 1 librados: calling monclient init 2024-03-29T10:58:19.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.834+0000 7fe8d8e25640 10 monclient: init 2024-03-29T10:58:19.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.835+0000 7fe8d8e25640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:19.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.835+0000 7fe8d8e25640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:58:19.818 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.835+0000 7fe8d8e25640 10 monclient(hunting): picked mon.noname-b con 0x7fe8d40a9c70 addr v1:172.21.15.118:6789/0 2024-03-29T10:58:19.818 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.837+0000 7fe8d8e25640 10 monclient(hunting): picked mon.noname-a con 0x7fe8d40b2520 addr v1:172.21.15.112:6789/0 2024-03-29T10:58:19.818 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.837+0000 7fe8d8e25640 10 monclient(hunting): picked mon.noname-c con 0x7fe8d413b460 addr v1:172.21.15.112:6790/0 2024-03-29T10:58:19.818 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.837+0000 7fe8d8e25640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d40abb30 con 0x7fe8d40b2520 2024-03-29T10:58:19.818 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.837+0000 7fe8d8e25640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d413f260 con 0x7fe8d40a9c70 2024-03-29T10:58:19.818 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.837+0000 7fe8d8e25640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d40abda0 con 0x7fe8d413b460 2024-03-29T10:58:19.818 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.837+0000 7fe8d8e25640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:19.818 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.837+0000 7fe8d8e25640 10 monclient(hunting): authenticate will time out at 1255.141821s 2024-03-29T10:58:19.822 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.842+0000 7fe8d259c640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fe8d40a9c70 0x7fe8d413e4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:44909/0 (socket says 172.21.15.112:44909) 2024-03-29T10:58:19.822 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.842+0000 7fe8d259c640 1 -- 172.21.15.112:0/213043900 learned_addr learned my addr 172.21.15.112:0/213043900 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:58:19.822 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.842+0000 7fe8c2ffd640 10 client.?.objecter ms_handle_connect 0x7fe8d40b2520 2024-03-29T10:58:19.822 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.842+0000 7fe8c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:19.824 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.844+0000 7fe8c2ffd640 10 client.?.objecter ms_handle_connect 0x7fe8d40a9c70 2024-03-29T10:58:19.825 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.844+0000 7fe8c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:19.829 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.845+0000 7fe8c2ffd640 10 client.?.objecter ms_handle_connect 0x7fe8d413b460 2024-03-29T10:58:19.829 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.845+0000 7fe8c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:19.829 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.845+0000 7fe8c2ffd640 1 -- 172.21.15.112:0/213043900 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2160831651 0 0) 0x7fe8d413f260 con 0x7fe8d40a9c70 2024-03-29T10:58:19.829 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.845+0000 7fe8c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:19.829 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.845+0000 7fe8c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:19.829 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.845+0000 7fe8c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:19.829 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.845+0000 7fe8c2ffd640 10 monclient(hunting): my global_id is 4468 2024-03-29T10:58:19.829 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.845+0000 7fe8c2ffd640 1 -- 172.21.15.112:0/213043900 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8b4004eb0 con 0x7fe8d40a9c70 2024-03-29T10:58:19.829 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.845+0000 7fe8c2ffd640 1 -- 172.21.15.112:0/213043900 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4018890032 0 0) 0x7fe8d40abda0 con 0x7fe8d413b460 2024-03-29T10:58:19.830 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.845+0000 7fe8c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:19.830 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.845+0000 7fe8c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:19.830 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.845+0000 7fe8c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:19.830 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.846+0000 7fe8c2ffd640 10 monclient(hunting): my global_id is 4502 2024-03-29T10:58:19.830 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.846+0000 7fe8c2ffd640 1 -- 172.21.15.112:0/213043900 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d413f260 con 0x7fe8d413b460 2024-03-29T10:58:19.830 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.846+0000 7fe8c2ffd640 1 -- 172.21.15.112:0/213043900 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1073299249 0 0) 0x7fe8b4004eb0 con 0x7fe8d40a9c70 2024-03-29T10:58:19.830 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.846+0000 7fe8c2ffd640 1 -- 172.21.15.112:0/213043900 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8d40abda0 con 0x7fe8d40a9c70 2024-03-29T10:58:19.830 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.846+0000 7fe8c2ffd640 1 -- 172.21.15.112:0/213043900 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe8c8001a30 con 0x7fe8d40a9c70 2024-03-29T10:58:19.830 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.846+0000 7fe8c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:19.830 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.846+0000 7fe8c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:19.831 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.846+0000 7fe8c2ffd640 10 monclient(hunting): dump: 2024-03-29T10:58:19.831 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:19.831 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:19.831 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:19.831 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:19.831 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:19.831 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:19.831 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:19.831 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:19.831 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:19.831 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:19.831 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.846+0000 7fe8c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:19.831 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.846+0000 7fe8c2ffd640 1 -- 172.21.15.112:0/213043900 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129304351 0 0) 0x7fe8d40abb30 con 0x7fe8d40b2520 2024-03-29T10:58:19.831 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.846+0000 7fe8c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:19.832 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.846+0000 7fe8c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:19.832 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.846+0000 7fe8c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:19.832 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.846+0000 7fe8c2ffd640 10 monclient(hunting): my global_id is 4473 2024-03-29T10:58:19.832 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.847+0000 7fe8c2ffd640 1 -- 172.21.15.112:0/213043900 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8b4004eb0 con 0x7fe8d40b2520 2024-03-29T10:58:19.832 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.847+0000 7fe8c2ffd640 1 -- 172.21.15.112:0/213043900 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 822385081 0 0) 0x7fe8d40abda0 con 0x7fe8d40a9c70 2024-03-29T10:58:19.832 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.847+0000 7fe8c2ffd640 1 -- 172.21.15.112:0/213043900 >> v1:172.21.15.112:6790/0 conn(0x7fe8d413b460 legacy=0x7fe8d41e1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:19.832 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.847+0000 7fe8c2ffd640 1 -- 172.21.15.112:0/213043900 >> v1:172.21.15.112:6789/0 conn(0x7fe8d40b2520 legacy=0x7fe8d41d6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:19.832 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.847+0000 7fe8c2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:58:19.832 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.847+0000 7fe8c2ffd640 1 monclient: found mon.b 2024-03-29T10:58:19.832 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.847+0000 7fe8c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:19.832 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.847+0000 7fe8c2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:58:19.832 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.847+0000 7fe8c2ffd640 1 -- 172.21.15.112:0/213043900 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d413ed40 con 0x7fe8d40a9c70 2024-03-29T10:58:19.832 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.847+0000 7fe8c2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:58:19.833 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.847+0000 7fe8c2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:58:19.833 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.847+0000 7fe8c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:19.833 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.848+0000 7fe8d8e25640 5 monclient: authenticate success, global_id 4468 2024-03-29T10:58:19.833 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.848+0000 7fe8d8e25640 10 monclient: _renew_subs 2024-03-29T10:58:19.833 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.848+0000 7fe8d8e25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:58:19.833 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.848+0000 7fe8d8e25640 1 -- 172.21.15.112:0/213043900 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8d41d7520 con 0x7fe8d40a9c70 2024-03-29T10:58:19.833 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.848+0000 7fe8d8e25640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:19.833 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.848+0000 7fe8d8e25640 10 monclient: _renew_subs 2024-03-29T10:58:19.833 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.848+0000 7fe8d8e25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:58:19.833 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.848+0000 7fe8d8e25640 1 -- 172.21.15.112:0/213043900 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8d41e3050 con 0x7fe8d40a9c70 2024-03-29T10:58:19.833 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.848+0000 7fe8d8e25640 1 librados: init done 2024-03-29T10:58:19.835 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.849+0000 7fe8c2ffd640 1 -- 172.21.15.112:0/213043900 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8c8001c70 con 0x7fe8d40a9c70 2024-03-29T10:58:19.835 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.849+0000 7fe8c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:19.835 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.849+0000 7fe8c2ffd640 1 -- 172.21.15.112:0/213043900 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe8c8003a60 con 0x7fe8d40a9c70 2024-03-29T10:58:19.835 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.849+0000 7fe8c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:19.835 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.849+0000 7fe8c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:19.835 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.849+0000 7fe8c2ffd640 10 monclient: dump: 2024-03-29T10:58:19.835 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:19.835 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:19.836 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:19.836 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:19.836 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:19.836 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:19.836 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:19.836 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:19.836 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:19.836 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:19.836 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.849+0000 7fe8d0d99640 4 set_mon_vals no callback set 2024-03-29T10:58:19.836 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.851+0000 7fe8c2ffd640 1 -- 172.21.15.112:0/213043900 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fe8c8003bc0 con 0x7fe8d40a9c70 2024-03-29T10:58:19.836 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.852+0000 7fe8d8e25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:19.836 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.852+0000 7fe8d8e25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:19.836 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.852+0000 7fe8d8e25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:58:19.836 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.852+0000 7fe8d8e25640 1 -- 172.21.15.112:0/213043900 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8b8005080 con 0x7fe8d40a9c70 2024-03-29T10:58:19.837 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.852+0000 7fe8c2ffd640 1 -- 172.21.15.112:0/213043900 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7fe8c8084ae0 con 0x7fe8d40a9c70 2024-03-29T10:58:19.837 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.852+0000 7fe8c2ffd640 10 client.4468.objecter ms_dispatch 0x7fe8d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:19.837 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.852+0000 7fe8c2ffd640 3 client.4468.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:19.837 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.852+0000 7fe8c2ffd640 3 client.4468.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:19.837 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.853+0000 7fe8c2ffd640 20 client.4468.objecter dump_active .. 0 homeless 2024-03-29T10:58:19.837 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.854+0000 7fe8d1d9b640 10 monclient: get_auth_request con 0x7fe8b406dc90 auth_method 0 2024-03-29T10:58:19.838 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.854+0000 7fe8c2ffd640 10 client.4468.objecter ms_handle_connect 0x7fe8b406dc90 2024-03-29T10:58:19.842 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.861+0000 7fe8c2ffd640 1 -- 172.21.15.112:0/213043900 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8c80503c0 con 0x7fe8d40a9c70 2024-03-29T10:58:19.842 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.862+0000 7fe8c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:19.842 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:19.862+0000 7fe8c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:20.161 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:20.181+0000 7fe8d8e25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:20.161 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:20.181+0000 7fe8d8e25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:20.161 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:20.181+0000 7fe8d8e25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:58:20.162 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:20.181+0000 7fe8d8e25640 1 -- 172.21.15.112:0/213043900 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8b8005330 con 0x7fe8d40a9c70 2024-03-29T10:58:20.162 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:20.182+0000 7fe8c2ffd640 1 -- 172.21.15.112:0/213043900 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7fe8c8059220 con 0x7fe8d40a9c70 2024-03-29T10:58:20.162 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:20.182+0000 7fe8c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:20.163 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:20.182+0000 7fe8c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:20.164 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:58:20.164 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:20.168 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:20.187+0000 7fe8d8e25640 10 librados: watch_flush enter 2024-03-29T10:58:20.168 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:20.187+0000 7fe8d8e25640 10 librados: watch_flush exit 2024-03-29T10:58:20.168 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:20.187+0000 7fe8d8e25640 20 client.4468.objecter shutdown clearing up homeless session... 2024-03-29T10:58:20.168 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:20.187+0000 7fe8d8e25640 10 client.4468.objecter successfully canceled tick 2024-03-29T10:58:20.168 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:20.187+0000 7fe8d8e25640 1 -- 172.21.15.112:0/213043900 >> v1:172.21.15.118:6816/3181073408 conn(0x7fe8b406dc90 legacy=0x7fe8b4070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:20.168 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:20.187+0000 7fe8d8e25640 10 monclient: shutdown 2024-03-29T10:58:20.168 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:20.187+0000 7fe8d8e25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:20.168 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:20.187+0000 7fe8d8e25640 1 -- 172.21.15.112:0/213043900 >> v1:172.21.15.118:6789/0 conn(0x7fe8d40a9c70 legacy=0x7fe8d413e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:20.168 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:20.188+0000 7fe8d8e25640 1 -- 172.21.15.112:0/213043900 shutdown_connections 2024-03-29T10:58:20.168 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:20.188+0000 7fe8d8e25640 1 -- 172.21.15.112:0/213043900 >> 172.21.15.112:0/213043900 conn(0x7fe8d406d180 msgr2=0x7fe8d4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:20.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:20.188+0000 7fe8d8e25640 1 -- 172.21.15.112:0/213043900 shutdown_connections 2024-03-29T10:58:20.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:20.188+0000 7fe8d8e25640 1 -- 172.21.15.112:0/213043900 wait complete. 2024-03-29T10:58:20.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:20.188+0000 7fe8d8e25640 1 librados: shutdown 2024-03-29T10:58:20.185 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:26.992 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:27.156 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.174+0000 7f7fadc4e640 10 monclient: get_monmap_and_config 2024-03-29T10:58:27.160 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.174+0000 7f7fadc4e640 10 monclient: build_initial_monmap 2024-03-29T10:58:27.160 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.174+0000 7f7fadc4e640 10 monclient: monmap: 2024-03-29T10:58:27.161 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:58:27.161 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:27.161 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:58:27.175484+0000 2024-03-29T10:58:27.161 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:58:27.175484+0000 2024-03-29T10:58:27.161 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:27.161 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:27.161 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:58:27.161 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:58:27.161 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:58:27.161 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:27.161 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.176+0000 7f7fadc4e640 1 Processor -- start 2024-03-29T10:58:27.163 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.179+0000 7f7fadc4e640 1 -- start start 2024-03-29T10:58:27.163 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.179+0000 7f7fadc4e640 10 monclient: init 2024-03-29T10:58:27.163 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.180+0000 7f7fadc4e640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:27.163 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.180+0000 7f7fadc4e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:58:27.164 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.180+0000 7f7fadc4e640 10 monclient(hunting): picked mon.noname-b con 0x7f7fa80a9ac0 addr v1:172.21.15.112:6790/0 2024-03-29T10:58:27.164 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.180+0000 7f7fadc4e640 10 monclient(hunting): picked mon.noname-c con 0x7f7fa813ee90 addr v1:172.21.15.118:6789/0 2024-03-29T10:58:27.164 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.180+0000 7f7fadc4e640 10 monclient(hunting): picked mon.noname-a con 0x7f7fa80af900 addr v1:172.21.15.112:6789/0 2024-03-29T10:58:27.164 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.180+0000 7f7fadc4e640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa80afd10 con 0x7f7fa80af900 2024-03-29T10:58:27.164 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.180+0000 7f7fadc4e640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa80af6f0 con 0x7f7fa813ee90 2024-03-29T10:58:27.164 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.180+0000 7f7fadc4e640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa80aac80 con 0x7f7fa80a9ac0 2024-03-29T10:58:27.164 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.180+0000 7f7fadc4e640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:27.164 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.180+0000 7f7fadc4e640 10 monclient(hunting): authenticate will time out at 1262.484720s 2024-03-29T10:58:27.164 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.183+0000 7f7fa77fe640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f7fa80a9ac0 0x7f7fa80adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:39886/0 (socket says 172.21.15.112:39886) 2024-03-29T10:58:27.164 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.183+0000 7f7fa77fe640 1 -- 172.21.15.112:0/2797601259 learned_addr learned my addr 172.21.15.112:0/2797601259 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:58:27.166 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.184+0000 7f7fa67fc640 1 -- 172.21.15.112:0/2797601259 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135896398 0 0) 0x7f7fa80afd10 con 0x7f7fa80af900 2024-03-29T10:58:27.166 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.184+0000 7f7fa67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:27.166 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.184+0000 7f7fa67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:27.166 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.184+0000 7f7fa67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:27.166 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.184+0000 7f7fa67fc640 10 monclient(hunting): my global_id is 4476 2024-03-29T10:58:27.166 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.184+0000 7f7fa67fc640 1 -- 172.21.15.112:0/2797601259 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f90003520 con 0x7f7fa80af900 2024-03-29T10:58:27.166 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.184+0000 7f7fa67fc640 1 -- 172.21.15.112:0/2797601259 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847101641 0 0) 0x7f7fa80aac80 con 0x7f7fa80a9ac0 2024-03-29T10:58:27.167 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.184+0000 7f7fa67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:27.167 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.184+0000 7f7fa67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:27.167 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.184+0000 7f7fa67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:27.167 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.184+0000 7f7fa67fc640 10 monclient(hunting): my global_id is 4505 2024-03-29T10:58:27.167 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.184+0000 7f7fa67fc640 1 -- 172.21.15.112:0/2797601259 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fa80afd10 con 0x7f7fa80a9ac0 2024-03-29T10:58:27.168 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.188+0000 7f7fa67fc640 1 -- 172.21.15.112:0/2797601259 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079305665 0 0) 0x7f7fa80af6f0 con 0x7f7fa813ee90 2024-03-29T10:58:27.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.188+0000 7f7fa67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:27.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.188+0000 7f7fa67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:27.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.188+0000 7f7fa67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:27.170 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.188+0000 7f7fa67fc640 10 monclient(hunting): my global_id is 4471 2024-03-29T10:58:27.170 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.188+0000 7f7fa67fc640 1 -- 172.21.15.112:0/2797601259 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fa80aac80 con 0x7f7fa813ee90 2024-03-29T10:58:27.170 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.188+0000 7f7fa67fc640 1 -- 172.21.15.112:0/2797601259 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2597201838 0 0) 0x7f7fa80afd10 con 0x7f7fa80a9ac0 2024-03-29T10:58:27.170 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.188+0000 7f7fa67fc640 1 -- 172.21.15.112:0/2797601259 >> v1:172.21.15.118:6789/0 conn(0x7f7fa813ee90 legacy=0x7f7fa813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:27.170 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.188+0000 7f7fa67fc640 1 -- 172.21.15.112:0/2797601259 >> v1:172.21.15.112:6789/0 conn(0x7f7fa80af900 legacy=0x7f7fa8149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:27.170 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.188+0000 7f7fa67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:58:27.170 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.188+0000 7f7fa67fc640 1 monclient: found mon.noname-b 2024-03-29T10:58:27.170 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.188+0000 7f7fa67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:27.171 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.188+0000 7f7fa67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:58:27.171 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.188+0000 7f7fa67fc640 1 -- 172.21.15.112:0/2797601259 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa80aaf60 con 0x7f7fa80a9ac0 2024-03-29T10:58:27.171 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.188+0000 7f7fa67fc640 10 monclient: _finish_auth 0 2024-03-29T10:58:27.171 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.188+0000 7f7fa67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:58:27.171 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.188+0000 7f7fa67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:27.171 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.188+0000 7f7fa67fc640 1 -- 172.21.15.112:0/2797601259 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f7fa0005010 con 0x7f7fa80a9ac0 2024-03-29T10:58:27.171 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.188+0000 7f7fa67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:27.171 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.188+0000 7f7fa67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:27.171 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.188+0000 7f7fa67fc640 10 monclient: dump: 2024-03-29T10:58:27.171 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:27.171 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:27.171 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:27.172 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:27.172 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:27.172 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:27.172 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:27.172 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:27.172 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:27.172 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:27.172 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.188+0000 7f7fadc4e640 5 monclient: authenticate success, global_id 4505 2024-03-29T10:58:27.172 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.188+0000 7f7fadc4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:27.176 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.196+0000 7f7fa67fc640 1 -- 172.21.15.112:0/2797601259 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fa0005170 con 0x7f7fa80a9ac0 2024-03-29T10:58:27.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.196+0000 7f7fa67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:27.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.196+0000 7f7fa67fc640 1 -- 172.21.15.112:0/2797601259 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f7fa0001770 con 0x7f7fa80a9ac0 2024-03-29T10:58:27.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.196+0000 7f7fa67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:27.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.196+0000 7f7fa67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:27.177 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.196+0000 7f7fa67fc640 10 monclient: dump: 2024-03-29T10:58:27.177 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:27.178 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:27.178 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:27.178 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:27.178 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:27.178 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:27.178 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:27.178 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:27.178 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:27.178 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:27.178 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.196+0000 7f7fadc4e640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:27.178 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.196+0000 7f7fadc4e640 4 set_mon_vals no callback set 2024-03-29T10:58:27.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.197+0000 7f7fadc4e640 10 monclient: shutdown 2024-03-29T10:58:27.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.197+0000 7f7fadc4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:27.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.197+0000 7f7fadc4e640 1 -- 172.21.15.112:0/2797601259 >> v1:172.21.15.112:6790/0 conn(0x7f7fa80a9ac0 legacy=0x7f7fa80adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:27.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.197+0000 7f7fadc4e640 1 -- 172.21.15.112:0/2797601259 shutdown_connections 2024-03-29T10:58:27.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.197+0000 7f7fadc4e640 1 -- 172.21.15.112:0/2797601259 >> 172.21.15.112:0/2797601259 conn(0x7f7fa80b35e0 msgr2=0x7f7fa80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:27.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.198+0000 7f7fadc4e640 1 -- 172.21.15.112:0/2797601259 shutdown_connections 2024-03-29T10:58:27.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.198+0000 7f7fadc4e640 1 -- 172.21.15.112:0/2797601259 wait complete. 2024-03-29T10:58:27.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.199+0000 7f7fadc4e640 10 monclient: build_initial_monmap 2024-03-29T10:58:27.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.199+0000 7f7fadc4e640 10 monclient: monmap: 2024-03-29T10:58:27.180 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:58:27.180 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:27.181 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:58:27.181 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:58:27.181 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:27.181 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:27.181 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:58:27.181 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:58:27.181 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:58:27.181 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:27.181 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.199+0000 7f7fadc4e640 1 librados: starting msgr at 2024-03-29T10:58:27.181 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.199+0000 7f7fadc4e640 1 librados: starting objecter 2024-03-29T10:58:27.181 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.199+0000 7f7fadc4e640 1 Processor -- start 2024-03-29T10:58:27.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.200+0000 7f7fadc4e640 1 -- start start 2024-03-29T10:58:27.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.200+0000 7f7fadc4e640 1 librados: setting wanted keys 2024-03-29T10:58:27.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.200+0000 7f7fadc4e640 1 librados: calling monclient init 2024-03-29T10:58:27.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.200+0000 7f7fadc4e640 10 monclient: init 2024-03-29T10:58:27.184 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.200+0000 7f7fadc4e640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:27.184 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.200+0000 7f7fadc4e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:58:27.184 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.200+0000 7f7fadc4e640 10 monclient(hunting): picked mon.noname-c con 0x7f7fa80af900 addr v1:172.21.15.112:6790/0 2024-03-29T10:58:27.184 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.200+0000 7f7fadc4e640 10 monclient(hunting): picked mon.noname-a con 0x7f7fa813ee90 addr v1:172.21.15.112:6789/0 2024-03-29T10:58:27.184 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.200+0000 7f7fadc4e640 10 monclient(hunting): picked mon.noname-b con 0x7f7fa806f7f0 addr v1:172.21.15.118:6789/0 2024-03-29T10:58:27.184 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.200+0000 7f7fadc4e640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa81314f0 con 0x7f7fa813ee90 2024-03-29T10:58:27.184 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.200+0000 7f7fadc4e640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa81db9f0 con 0x7f7fa806f7f0 2024-03-29T10:58:27.184 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.200+0000 7f7fadc4e640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa81dc620 con 0x7f7fa80af900 2024-03-29T10:58:27.184 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.200+0000 7f7fadc4e640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:27.184 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.200+0000 7f7fadc4e640 10 monclient(hunting): authenticate will time out at 1262.504949s 2024-03-29T10:58:27.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.204+0000 7f7fa7fff640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f7fa806f7f0 0x7f7fa81e6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:45439/0 (socket says 172.21.15.112:45439) 2024-03-29T10:58:27.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.204+0000 7f7fa7fff640 1 -- 172.21.15.112:0/3034629836 learned_addr learned my addr 172.21.15.112:0/3034629836 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:58:27.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.205+0000 7f7f8ffff640 10 client.?.objecter ms_handle_connect 0x7f7fa806f7f0 2024-03-29T10:58:27.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.205+0000 7f7f8ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:27.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.205+0000 7f7f8ffff640 1 -- 172.21.15.112:0/3034629836 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454464652 0 0) 0x7f7fa81db9f0 con 0x7f7fa806f7f0 2024-03-29T10:58:27.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.205+0000 7f7f8ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:27.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.205+0000 7f7f8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:27.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.205+0000 7f7f8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:27.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.205+0000 7f7f8ffff640 10 monclient(hunting): my global_id is 4474 2024-03-29T10:58:27.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.205+0000 7f7f8ffff640 1 -- 172.21.15.112:0/3034629836 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f94004eb0 con 0x7f7fa806f7f0 2024-03-29T10:58:27.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.205+0000 7f7f8ffff640 1 -- 172.21.15.112:0/3034629836 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3481603469 0 0) 0x7f7f94004eb0 con 0x7f7fa806f7f0 2024-03-29T10:58:27.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.205+0000 7f7f8ffff640 1 -- 172.21.15.112:0/3034629836 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fa81db9f0 con 0x7f7fa806f7f0 2024-03-29T10:58:27.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.205+0000 7f7f8ffff640 1 -- 172.21.15.112:0/3034629836 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f7f980018e0 con 0x7f7fa806f7f0 2024-03-29T10:58:27.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.205+0000 7f7f8ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:27.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.205+0000 7f7f8ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:27.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.205+0000 7f7f8ffff640 10 monclient(hunting): dump: 2024-03-29T10:58:27.187 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:27.187 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:27.187 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:27.187 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:27.187 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:27.187 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:27.187 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:27.188 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:27.188 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:27.188 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:27.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.205+0000 7f7f8ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:27.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.206+0000 7f7f8ffff640 1 -- 172.21.15.112:0/3034629836 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4003816046 0 0) 0x7f7fa81db9f0 con 0x7f7fa806f7f0 2024-03-29T10:58:27.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.206+0000 7f7f8ffff640 1 -- 172.21.15.112:0/3034629836 >> v1:172.21.15.112:6790/0 conn(0x7f7fa80af900 legacy=0x7f7fa8130020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:27.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.206+0000 7f7f8ffff640 1 -- 172.21.15.112:0/3034629836 >> v1:172.21.15.112:6789/0 conn(0x7f7fa813ee90 legacy=0x7f7fa81dbf10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:58:27.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.207+0000 7f7f8ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:58:27.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.207+0000 7f7f8ffff640 1 monclient: found mon.b 2024-03-29T10:58:27.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.207+0000 7f7f8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:27.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.207+0000 7f7f8ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:58:27.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.207+0000 7f7f8ffff640 1 -- 172.21.15.112:0/3034629836 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa8071c90 con 0x7f7fa806f7f0 2024-03-29T10:58:27.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.207+0000 7f7f8ffff640 10 monclient: _finish_auth 0 2024-03-29T10:58:27.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.207+0000 7f7f8ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:58:27.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.207+0000 7f7f8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:27.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.207+0000 7f7fadc4e640 5 monclient: authenticate success, global_id 4474 2024-03-29T10:58:27.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.207+0000 7f7fadc4e640 10 monclient: _renew_subs 2024-03-29T10:58:27.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.207+0000 7f7fadc4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:58:27.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.207+0000 7f7fadc4e640 1 -- 172.21.15.112:0/3034629836 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fa8071f40 con 0x7f7fa806f7f0 2024-03-29T10:58:27.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.207+0000 7f7fadc4e640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:27.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.207+0000 7f7fadc4e640 10 monclient: _renew_subs 2024-03-29T10:58:27.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.207+0000 7f7fadc4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:58:27.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.207+0000 7f7fadc4e640 1 -- 172.21.15.112:0/3034629836 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa81e7b90 con 0x7f7fa806f7f0 2024-03-29T10:58:27.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.207+0000 7f7fadc4e640 1 librados: init done 2024-03-29T10:58:27.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.208+0000 7f7fadc4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:27.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.208+0000 7f7fadc4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:27.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.208+0000 7f7fadc4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:58:27.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.208+0000 7f7fadc4e640 1 -- 172.21.15.112:0/3034629836 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa80aeaa0 con 0x7f7fa806f7f0 2024-03-29T10:58:27.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.209+0000 7f7f8ffff640 1 -- 172.21.15.112:0/3034629836 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f98003610 con 0x7f7fa806f7f0 2024-03-29T10:58:27.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.209+0000 7f7f8ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:27.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.209+0000 7f7f8ffff640 1 -- 172.21.15.112:0/3034629836 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f7f98003a20 con 0x7f7fa806f7f0 2024-03-29T10:58:27.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.209+0000 7f7f8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:27.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.209+0000 7f7f8ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:27.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.209+0000 7f7f8ffff640 10 monclient: dump: 2024-03-29T10:58:27.191 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:27.192 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:27.192 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:27.192 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:27.192 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:27.192 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:27.192 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:27.192 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:27.192 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:27.192 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:27.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.209+0000 7f7fa5ffb640 4 set_mon_vals no callback set 2024-03-29T10:58:27.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.211+0000 7f7f8ffff640 1 -- 172.21.15.112:0/3034629836 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f7f98003c40 con 0x7f7fa806f7f0 2024-03-29T10:58:27.194 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.211+0000 7f7f8ffff640 1 -- 172.21.15.112:0/3034629836 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f7f98054850 con 0x7f7fa806f7f0 2024-03-29T10:58:27.194 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.211+0000 7f7f8ffff640 10 client.4474.objecter ms_dispatch 0x7f7fa8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:27.194 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.211+0000 7f7f8ffff640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:27.195 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.211+0000 7f7f8ffff640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:27.195 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.211+0000 7f7f8ffff640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-29T10:58:27.195 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.213+0000 7f7fa77fe640 10 monclient: get_auth_request con 0x7f7f9406db70 auth_method 0 2024-03-29T10:58:27.195 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.214+0000 7f7f8ffff640 10 client.4474.objecter ms_handle_connect 0x7f7f9406db70 2024-03-29T10:58:27.195 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.215+0000 7f7f8ffff640 1 -- 172.21.15.112:0/3034629836 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7f98050af0 con 0x7f7fa806f7f0 2024-03-29T10:58:27.195 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.215+0000 7f7f8ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:27.195 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.215+0000 7f7f8ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:27.562 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.582+0000 7f7fadc4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:27.562 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.582+0000 7f7fadc4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:27.563 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.582+0000 7f7fadc4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:58:27.563 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.582+0000 7f7fadc4e640 1 -- 172.21.15.112:0/3034629836 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fa81db850 con 0x7f7fa806f7f0 2024-03-29T10:58:27.563 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.583+0000 7f7f8ffff640 1 -- 172.21.15.112:0/3034629836 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f7f98050af0 con 0x7f7fa806f7f0 2024-03-29T10:58:27.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.583+0000 7f7f8ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:27.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.583+0000 7f7f8ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:27.564 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:58:27.564 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:27.567 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.586+0000 7f7fadc4e640 10 librados: watch_flush enter 2024-03-29T10:58:27.567 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.586+0000 7f7fadc4e640 10 librados: watch_flush exit 2024-03-29T10:58:27.567 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.586+0000 7f7fadc4e640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-29T10:58:27.567 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.586+0000 7f7fadc4e640 10 client.4474.objecter successfully canceled tick 2024-03-29T10:58:27.568 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.586+0000 7f7fadc4e640 1 -- 172.21.15.112:0/3034629836 >> v1:172.21.15.118:6816/3181073408 conn(0x7f7f9406db70 legacy=0x7f7f94070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:27.568 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.586+0000 7f7fadc4e640 10 monclient: shutdown 2024-03-29T10:58:27.568 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.586+0000 7f7fadc4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:27.568 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.586+0000 7f7fadc4e640 1 -- 172.21.15.112:0/3034629836 >> v1:172.21.15.118:6789/0 conn(0x7f7fa806f7f0 legacy=0x7f7fa81e6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:27.568 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.587+0000 7f7fadc4e640 1 -- 172.21.15.112:0/3034629836 shutdown_connections 2024-03-29T10:58:27.568 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.587+0000 7f7fadc4e640 1 -- 172.21.15.112:0/3034629836 >> 172.21.15.112:0/3034629836 conn(0x7f7fa80b35e0 msgr2=0x7f7fa80adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:27.571 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.587+0000 7f7fadc4e640 1 -- 172.21.15.112:0/3034629836 shutdown_connections 2024-03-29T10:58:27.571 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.591+0000 7f7fadc4e640 1 -- 172.21.15.112:0/3034629836 wait complete. 2024-03-29T10:58:27.573 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:27.592+0000 7f7fadc4e640 1 librados: shutdown 2024-03-29T10:58:27.593 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:34.400 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:34.548 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.566+0000 7fdad5851640 10 monclient: get_monmap_and_config 2024-03-29T10:58:34.548 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.566+0000 7fdad5851640 10 monclient: build_initial_monmap 2024-03-29T10:58:34.548 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.566+0000 7fdad5851640 10 monclient: monmap: 2024-03-29T10:58:34.548 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:58:34.548 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:34.548 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:58:34.567614+0000 2024-03-29T10:58:34.548 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:58:34.567614+0000 2024-03-29T10:58:34.549 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:34.549 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:34.549 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:58:34.549 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:58:34.549 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:58:34.549 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:34.549 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.568+0000 7fdad5851640 1 Processor -- start 2024-03-29T10:58:34.549 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.568+0000 7fdad5851640 1 -- start start 2024-03-29T10:58:34.549 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.568+0000 7fdad5851640 10 monclient: init 2024-03-29T10:58:34.549 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.569+0000 7fdad5851640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:34.550 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.569+0000 7fdad5851640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:58:34.550 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.569+0000 7fdad5851640 10 monclient(hunting): picked mon.noname-a con 0x7fdac8034850 addr v1:172.21.15.112:6789/0 2024-03-29T10:58:34.550 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.569+0000 7fdad5851640 10 monclient(hunting): picked mon.noname-b con 0x7fdac802a640 addr v1:172.21.15.112:6790/0 2024-03-29T10:58:34.550 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.569+0000 7fdad5851640 10 monclient(hunting): picked mon.noname-c con 0x7fdac80327d0 addr v1:172.21.15.118:6789/0 2024-03-29T10:58:34.550 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.569+0000 7fdad5851640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac802c820 con 0x7fdac8034850 2024-03-29T10:58:34.550 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.569+0000 7fdad5851640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac802a120 con 0x7fdac80327d0 2024-03-29T10:58:34.550 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.569+0000 7fdad5851640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac80c75b0 con 0x7fdac802a640 2024-03-29T10:58:34.550 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.569+0000 7fdad5851640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:34.550 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.569+0000 7fdad5851640 10 monclient(hunting): authenticate will time out at 1269.873608s 2024-03-29T10:58:34.555 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.571+0000 7fdacffff640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fdac802a640 0x7fdac80320c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:43806/0 (socket says 172.21.15.112:43806) 2024-03-29T10:58:34.555 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.571+0000 7fdacffff640 1 -- 172.21.15.112:0/3043291358 learned_addr learned my addr 172.21.15.112:0/3043291358 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:58:34.555 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.572+0000 7fdacf7fe640 1 -- 172.21.15.112:0/3043291358 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4221097428 0 0) 0x7fdac80c75b0 con 0x7fdac802a640 2024-03-29T10:58:34.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.572+0000 7fdacf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:34.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.572+0000 7fdacf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:34.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.572+0000 7fdacf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:34.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.572+0000 7fdacf7fe640 10 monclient(hunting): my global_id is 4508 2024-03-29T10:58:34.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.572+0000 7fdacf7fe640 1 -- 172.21.15.112:0/3043291358 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdac0003520 con 0x7fdac802a640 2024-03-29T10:58:34.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.572+0000 7fdacf7fe640 1 -- 172.21.15.112:0/3043291358 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 751143012 0 0) 0x7fdac802c820 con 0x7fdac8034850 2024-03-29T10:58:34.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.572+0000 7fdacf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:34.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.572+0000 7fdacf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:34.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.572+0000 7fdacf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:34.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.572+0000 7fdacf7fe640 10 monclient(hunting): my global_id is 4479 2024-03-29T10:58:34.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.572+0000 7fdacf7fe640 1 -- 172.21.15.112:0/3043291358 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdac80c75b0 con 0x7fdac8034850 2024-03-29T10:58:34.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.572+0000 7fdacf7fe640 1 -- 172.21.15.112:0/3043291358 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3910592233 0 0) 0x7fdac0003520 con 0x7fdac802a640 2024-03-29T10:58:34.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.572+0000 7fdacf7fe640 1 -- 172.21.15.112:0/3043291358 >> v1:172.21.15.118:6789/0 conn(0x7fdac80327d0 legacy=0x7fdac80cf240 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:58:34.556 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.572+0000 7fdacf7fe640 1 -- 172.21.15.112:0/3043291358 >> v1:172.21.15.112:6789/0 conn(0x7fdac8034850 legacy=0x7fdac80c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:34.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.572+0000 7fdacf7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:58:34.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.572+0000 7fdacf7fe640 1 monclient: found mon.noname-b 2024-03-29T10:58:34.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.572+0000 7fdacf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:34.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.572+0000 7fdacf7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:58:34.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.572+0000 7fdacf7fe640 1 -- 172.21.15.112:0/3043291358 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac8033110 con 0x7fdac802a640 2024-03-29T10:58:34.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.572+0000 7fdacf7fe640 10 monclient: _finish_auth 0 2024-03-29T10:58:34.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.572+0000 7fdacf7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:58:34.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.572+0000 7fdacf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:34.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.572+0000 7fdacf7fe640 1 -- 172.21.15.112:0/3043291358 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fdad004f730 con 0x7fdac802a640 2024-03-29T10:58:34.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.573+0000 7fdacf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:34.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.573+0000 7fdacf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:34.557 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.573+0000 7fdacf7fe640 10 monclient: dump: 2024-03-29T10:58:34.557 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:34.557 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:34.558 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:34.558 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:34.558 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:34.558 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:34.558 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:34.558 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:34.558 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:34.558 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:34.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.573+0000 7fdad5851640 5 monclient: authenticate success, global_id 4508 2024-03-29T10:58:34.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.573+0000 7fdad5851640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:34.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.573+0000 7fdacf7fe640 1 -- 172.21.15.112:0/3043291358 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdad0075f10 con 0x7fdac802a640 2024-03-29T10:58:34.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.573+0000 7fdacf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:34.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.573+0000 7fdad5851640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:34.558 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.573+0000 7fdad5851640 4 set_mon_vals no callback set 2024-03-29T10:58:34.559 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.573+0000 7fdacf7fe640 1 -- 172.21.15.112:0/3043291358 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fdad00758b0 con 0x7fdac802a640 2024-03-29T10:58:34.559 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.573+0000 7fdacf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:34.559 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.573+0000 7fdacf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:34.559 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.573+0000 7fdacf7fe640 10 monclient: dump: 2024-03-29T10:58:34.559 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:34.559 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:34.559 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:34.559 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:34.559 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:34.559 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:34.559 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:34.559 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:34.559 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:34.559 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:34.559 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.574+0000 7fdad5851640 10 monclient: shutdown 2024-03-29T10:58:34.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.574+0000 7fdad5851640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:34.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.574+0000 7fdad5851640 1 -- 172.21.15.112:0/3043291358 >> v1:172.21.15.112:6790/0 conn(0x7fdac802a640 legacy=0x7fdac80320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:34.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.575+0000 7fdad5851640 1 -- 172.21.15.112:0/3043291358 shutdown_connections 2024-03-29T10:58:34.560 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.575+0000 7fdad5851640 1 -- 172.21.15.112:0/3043291358 >> 172.21.15.112:0/3043291358 conn(0x7fdac80058f0 msgr2=0x7fdac8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:34.562 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.576+0000 7fdad5851640 1 -- 172.21.15.112:0/3043291358 shutdown_connections 2024-03-29T10:58:34.562 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.576+0000 7fdad5851640 1 -- 172.21.15.112:0/3043291358 wait complete. 2024-03-29T10:58:34.562 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.577+0000 7fdad5851640 10 monclient: build_initial_monmap 2024-03-29T10:58:34.562 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.577+0000 7fdad5851640 10 monclient: monmap: 2024-03-29T10:58:34.563 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:58:34.563 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:34.563 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:58:34.563 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:58:34.563 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:34.563 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:34.563 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:58:34.563 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:58:34.563 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:58:34.563 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:34.563 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.577+0000 7fdad5851640 1 librados: starting msgr at 2024-03-29T10:58:34.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.577+0000 7fdad5851640 1 librados: starting objecter 2024-03-29T10:58:34.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.577+0000 7fdad5851640 1 Processor -- start 2024-03-29T10:58:34.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.578+0000 7fdad5851640 1 -- start start 2024-03-29T10:58:34.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.578+0000 7fdad5851640 1 librados: setting wanted keys 2024-03-29T10:58:34.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.578+0000 7fdad5851640 1 librados: calling monclient init 2024-03-29T10:58:34.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.578+0000 7fdad5851640 10 monclient: init 2024-03-29T10:58:34.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.579+0000 7fdad5851640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:34.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.579+0000 7fdad5851640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:58:34.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.579+0000 7fdad5851640 10 monclient(hunting): picked mon.noname-a con 0x7fdac802a640 addr v1:172.21.15.112:6789/0 2024-03-29T10:58:34.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.579+0000 7fdad5851640 10 monclient(hunting): picked mon.noname-b con 0x7fdac80327d0 addr v1:172.21.15.118:6789/0 2024-03-29T10:58:34.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.579+0000 7fdad5851640 10 monclient(hunting): picked mon.noname-c con 0x7fdac8034850 addr v1:172.21.15.112:6790/0 2024-03-29T10:58:34.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.579+0000 7fdad5851640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac80c3a60 con 0x7fdac802a640 2024-03-29T10:58:34.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.579+0000 7fdad5851640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac8169a80 con 0x7fdac80327d0 2024-03-29T10:58:34.564 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.579+0000 7fdad5851640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac80c3780 con 0x7fdac8034850 2024-03-29T10:58:34.565 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.579+0000 7fdad5851640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:34.565 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.579+0000 7fdad5851640 10 monclient(hunting): authenticate will time out at 1269.883510s 2024-03-29T10:58:34.565 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.580+0000 7fdad5050640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fdac8034850 0x7fdac80c0260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:44682/0 (socket says 172.21.15.112:44682) 2024-03-29T10:58:34.565 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.580+0000 7fdad5050640 1 -- 172.21.15.112:0/1157272476 learned_addr learned my addr 172.21.15.112:0/1157272476 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:58:34.565 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.581+0000 7fdacd7fa640 10 client.?.objecter ms_handle_connect 0x7fdac8034850 2024-03-29T10:58:34.565 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.581+0000 7fdacd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:34.565 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.581+0000 7fdacd7fa640 1 -- 172.21.15.112:0/1157272476 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769219565 0 0) 0x7fdac80c3780 con 0x7fdac8034850 2024-03-29T10:58:34.565 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.581+0000 7fdacd7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:58:34.565 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.581+0000 7fdacd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:34.565 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.581+0000 7fdacd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:34.565 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.581+0000 7fdacd7fa640 10 monclient(hunting): my global_id is 4511 2024-03-29T10:58:34.565 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.581+0000 7fdacd7fa640 1 -- 172.21.15.112:0/1157272476 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdab4004eb0 con 0x7fdac8034850 2024-03-29T10:58:34.565 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.582+0000 7fdacd7fa640 1 -- 172.21.15.112:0/1157272476 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 269044317 0 0) 0x7fdab4004eb0 con 0x7fdac8034850 2024-03-29T10:58:34.566 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.582+0000 7fdacd7fa640 1 -- 172.21.15.112:0/1157272476 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdac80c3780 con 0x7fdac8034850 2024-03-29T10:58:34.566 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.582+0000 7fdacd7fa640 1 -- 172.21.15.112:0/1157272476 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fdac40017f0 con 0x7fdac8034850 2024-03-29T10:58:34.566 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.582+0000 7fdacd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:34.566 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.582+0000 7fdacd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:34.566 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.582+0000 7fdacd7fa640 10 monclient(hunting): dump: 2024-03-29T10:58:34.566 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:34.566 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:34.566 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:34.566 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:34.566 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:34.566 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:34.566 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:34.566 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:34.566 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:34.567 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:34.567 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.582+0000 7fdacd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:34.571 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.583+0000 7fdacd7fa640 10 client.?.objecter ms_handle_connect 0x7fdac802a640 2024-03-29T10:58:34.571 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.583+0000 7fdacd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:34.572 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.583+0000 7fdacd7fa640 1 -- 172.21.15.112:0/1157272476 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4112590265 0 0) 0x7fdac80c3780 con 0x7fdac8034850 2024-03-29T10:58:34.572 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.583+0000 7fdacd7fa640 1 -- 172.21.15.112:0/1157272476 >> v1:172.21.15.118:6789/0 conn(0x7fdac80327d0 legacy=0x7fdac80c3070 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:58:34.572 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.583+0000 7fdacd7fa640 1 -- 172.21.15.112:0/1157272476 >> v1:172.21.15.112:6789/0 conn(0x7fdac802a640 legacy=0x7fdac8169fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:34.572 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.583+0000 7fdacd7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:58:34.572 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.583+0000 7fdacd7fa640 1 monclient: found mon.c 2024-03-29T10:58:34.572 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.583+0000 7fdacd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:34.572 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.583+0000 7fdacd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:58:34.572 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.583+0000 7fdacd7fa640 1 -- 172.21.15.112:0/1157272476 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac8169710 con 0x7fdac8034850 2024-03-29T10:58:34.572 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.583+0000 7fdacd7fa640 10 monclient: _finish_auth 0 2024-03-29T10:58:34.572 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.583+0000 7fdacd7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:58:34.572 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.583+0000 7fdacd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:34.572 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.583+0000 7fdad5851640 5 monclient: authenticate success, global_id 4511 2024-03-29T10:58:34.572 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.583+0000 7fdad5851640 10 monclient: _renew_subs 2024-03-29T10:58:34.572 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.583+0000 7fdad5851640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:58:34.573 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.583+0000 7fdad5851640 1 -- 172.21.15.112:0/1157272476 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdac80c24f0 con 0x7fdac8034850 2024-03-29T10:58:34.573 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.583+0000 7fdad5851640 10 client.4511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:34.573 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.583+0000 7fdad5851640 10 monclient: _renew_subs 2024-03-29T10:58:34.573 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.583+0000 7fdad5851640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:58:34.573 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.583+0000 7fdad5851640 1 -- 172.21.15.112:0/1157272476 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdac814e2d0 con 0x7fdac8034850 2024-03-29T10:58:34.573 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.583+0000 7fdad5851640 1 librados: init done 2024-03-29T10:58:34.573 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.584+0000 7fdacd7fa640 1 -- 172.21.15.112:0/1157272476 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdac4001dc0 con 0x7fdac8034850 2024-03-29T10:58:34.573 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.584+0000 7fdacd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:34.573 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.584+0000 7fdaceffd640 4 set_mon_vals no callback set 2024-03-29T10:58:34.573 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.584+0000 7fdacd7fa640 1 -- 172.21.15.112:0/1157272476 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fdac4004a70 con 0x7fdac8034850 2024-03-29T10:58:34.573 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.584+0000 7fdacd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:34.573 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.584+0000 7fdacd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:34.573 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.584+0000 7fdacd7fa640 10 monclient: dump: 2024-03-29T10:58:34.573 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:34.573 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:34.574 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:34.574 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:34.574 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:34.574 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:34.574 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:34.574 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:34.574 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:34.574 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:34.574 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.585+0000 7fdad5851640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:34.574 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.585+0000 7fdad5851640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:34.575 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.585+0000 7fdad5851640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:58:34.575 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.585+0000 7fdad5851640 1 -- 172.21.15.112:0/1157272476 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdac8004cd0 con 0x7fdac8034850 2024-03-29T10:58:34.575 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.586+0000 7fdacd7fa640 1 -- 172.21.15.112:0/1157272476 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fdac401aac0 con 0x7fdac8034850 2024-03-29T10:58:34.575 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.587+0000 7fdacd7fa640 1 -- 172.21.15.112:0/1157272476 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7fdac40848e0 con 0x7fdac8034850 2024-03-29T10:58:34.575 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.587+0000 7fdacd7fa640 10 client.4511.objecter ms_dispatch 0x7fdad0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:34.575 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.587+0000 7fdacd7fa640 3 client.4511.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:34.575 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.587+0000 7fdacd7fa640 3 client.4511.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:34.575 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.587+0000 7fdacd7fa640 20 client.4511.objecter dump_active .. 0 homeless 2024-03-29T10:58:34.575 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.588+0000 7fdad484f640 10 monclient: get_auth_request con 0x7fdab406ddd0 auth_method 0 2024-03-29T10:58:34.575 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.588+0000 7fdacd7fa640 10 client.4511.objecter ms_handle_connect 0x7fdab406ddd0 2024-03-29T10:58:34.576 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.594+0000 7fdacd7fa640 1 -- 172.21.15.112:0/1157272476 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdac4050220 con 0x7fdac8034850 2024-03-29T10:58:34.576 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.594+0000 7fdacd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:34.576 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.594+0000 7fdacd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:34.900 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.919+0000 7fdad5851640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:34.900 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.919+0000 7fdad5851640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:34.900 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.919+0000 7fdad5851640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:58:34.900 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.919+0000 7fdad5851640 1 -- 172.21.15.112:0/1157272476 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdac81698a0 con 0x7fdac8034850 2024-03-29T10:58:34.902 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.921+0000 7fdacd7fa640 1 -- 172.21.15.112:0/1157272476 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7fdac4050220 con 0x7fdac8034850 2024-03-29T10:58:34.902 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.921+0000 7fdacd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:34.902 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.921+0000 7fdacd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:34.902 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:58:34.902 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:34.905 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.925+0000 7fdad5851640 10 librados: watch_flush enter 2024-03-29T10:58:34.906 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.926+0000 7fdad5851640 10 librados: watch_flush exit 2024-03-29T10:58:34.906 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.926+0000 7fdad5851640 20 client.4511.objecter shutdown clearing up homeless session... 2024-03-29T10:58:34.906 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.926+0000 7fdad5851640 10 client.4511.objecter successfully canceled tick 2024-03-29T10:58:34.906 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.926+0000 7fdad5851640 1 -- 172.21.15.112:0/1157272476 >> v1:172.21.15.118:6816/3181073408 conn(0x7fdab406ddd0 legacy=0x7fdab4070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:34.907 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.926+0000 7fdad5851640 10 monclient: shutdown 2024-03-29T10:58:34.907 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.926+0000 7fdad5851640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:34.907 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.926+0000 7fdad5851640 1 -- 172.21.15.112:0/1157272476 >> v1:172.21.15.112:6790/0 conn(0x7fdac8034850 legacy=0x7fdac80c0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:34.907 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.926+0000 7fdad5851640 1 -- 172.21.15.112:0/1157272476 shutdown_connections 2024-03-29T10:58:34.907 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.926+0000 7fdad5851640 1 -- 172.21.15.112:0/1157272476 >> 172.21.15.112:0/1157272476 conn(0x7fdac80058f0 msgr2=0x7fdac80043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:34.907 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.927+0000 7fdad5851640 1 -- 172.21.15.112:0/1157272476 shutdown_connections 2024-03-29T10:58:34.907 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.927+0000 7fdad5851640 1 -- 172.21.15.112:0/1157272476 wait complete. 2024-03-29T10:58:34.908 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:34.928+0000 7fdad5851640 1 librados: shutdown 2024-03-29T10:58:34.926 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:41.733 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:41.867 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.885+0000 7f5f0592e640 10 monclient: get_monmap_and_config 2024-03-29T10:58:41.867 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.885+0000 7f5f0592e640 10 monclient: build_initial_monmap 2024-03-29T10:58:41.868 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.885+0000 7f5f0592e640 10 monclient: monmap: 2024-03-29T10:58:41.868 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:58:41.868 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:41.868 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:58:41.887179+0000 2024-03-29T10:58:41.868 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:58:41.887179+0000 2024-03-29T10:58:41.868 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:41.868 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:41.868 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:58:41.868 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:58:41.868 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:58:41.868 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:41.868 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.886+0000 7f5f0592e640 1 Processor -- start 2024-03-29T10:58:41.868 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.886+0000 7f5f0592e640 1 -- start start 2024-03-29T10:58:41.868 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.886+0000 7f5f0592e640 10 monclient: init 2024-03-29T10:58:41.869 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.886+0000 7f5f0592e640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:41.869 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.886+0000 7f5f0592e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:58:41.869 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.886+0000 7f5f0592e640 10 monclient(hunting): picked mon.noname-c con 0x7f5f000ae250 addr v1:172.21.15.118:6789/0 2024-03-29T10:58:41.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.886+0000 7f5f0592e640 10 monclient(hunting): picked mon.noname-a con 0x7f5f0013f970 addr v1:172.21.15.112:6789/0 2024-03-29T10:58:41.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.886+0000 7f5f0592e640 10 monclient(hunting): picked mon.noname-b con 0x7f5f00142800 addr v1:172.21.15.112:6790/0 2024-03-29T10:58:41.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.886+0000 7f5f0592e640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f000afe00 con 0x7f5f0013f970 2024-03-29T10:58:41.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.886+0000 7f5f0592e640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f00142c10 con 0x7f5f000ae250 2024-03-29T10:58:41.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.886+0000 7f5f0592e640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f000afb20 con 0x7f5f00142800 2024-03-29T10:58:41.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.886+0000 7f5f0592e640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:41.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.886+0000 7f5f0592e640 10 monclient(hunting): authenticate will time out at 1277.192026s 2024-03-29T10:58:41.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.886+0000 7f5efe7fc640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5f0013f970 0x7f5f001420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:49606/0 (socket says 172.21.15.112:49606) 2024-03-29T10:58:41.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.886+0000 7f5efe7fc640 1 -- 172.21.15.112:0/1242616531 learned_addr learned my addr 172.21.15.112:0/1242616531 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:58:41.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.887+0000 7f5efdffb640 1 -- 172.21.15.112:0/1242616531 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 85144118 0 0) 0x7f5f000afe00 con 0x7f5f0013f970 2024-03-29T10:58:41.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.887+0000 7f5efdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:58:41.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.887+0000 7f5efdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:41.870 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.887+0000 7f5efdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:41.871 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.887+0000 7f5efdffb640 10 monclient(hunting): my global_id is 4485 2024-03-29T10:58:41.871 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.887+0000 7f5efdffb640 1 -- 172.21.15.112:0/1242616531 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ee8003520 con 0x7f5f0013f970 2024-03-29T10:58:41.871 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.887+0000 7f5efdffb640 1 -- 172.21.15.112:0/1242616531 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524821266 0 0) 0x7f5f000afb20 con 0x7f5f00142800 2024-03-29T10:58:41.871 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.887+0000 7f5efdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:58:41.871 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.887+0000 7f5efdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:41.871 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.887+0000 7f5efdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:41.871 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.887+0000 7f5efdffb640 10 monclient(hunting): my global_id is 4514 2024-03-29T10:58:41.871 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.887+0000 7f5efdffb640 1 -- 172.21.15.112:0/1242616531 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f000afe00 con 0x7f5f00142800 2024-03-29T10:58:41.871 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.887+0000 7f5efdffb640 1 -- 172.21.15.112:0/1242616531 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 132469416 0 0) 0x7f5f000afe00 con 0x7f5f00142800 2024-03-29T10:58:41.871 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.887+0000 7f5efdffb640 1 -- 172.21.15.112:0/1242616531 >> v1:172.21.15.118:6789/0 conn(0x7f5f000ae250 legacy=0x7f5f0013f260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:58:41.872 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.890+0000 7f5efdffb640 1 -- 172.21.15.112:0/1242616531 >> v1:172.21.15.112:6789/0 conn(0x7f5f0013f970 legacy=0x7f5f001420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:41.872 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.890+0000 7f5efdffb640 10 monclient: _finish_hunting 0 2024-03-29T10:58:41.872 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.890+0000 7f5efdffb640 1 monclient: found mon.noname-b 2024-03-29T10:58:41.872 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.890+0000 7f5efdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:41.872 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.890+0000 7f5efdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:58:41.872 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.890+0000 7f5efdffb640 1 -- 172.21.15.112:0/1242616531 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f000aab80 con 0x7f5f00142800 2024-03-29T10:58:41.872 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.890+0000 7f5efdffb640 10 monclient: _finish_auth 0 2024-03-29T10:58:41.872 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.890+0000 7f5efdffb640 10 monclient: _check_auth_tickets 2024-03-29T10:58:41.872 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.890+0000 7f5efdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:41.873 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.890+0000 7f5efdffb640 1 -- 172.21.15.112:0/1242616531 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f5ef0001670 con 0x7f5f00142800 2024-03-29T10:58:41.873 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.890+0000 7f5efdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:41.873 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.890+0000 7f5efdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:41.873 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.890+0000 7f5efdffb640 10 monclient: dump: 2024-03-29T10:58:41.873 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:41.873 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:41.873 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:41.873 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:41.873 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:41.873 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:41.873 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:41.873 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:41.873 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:41.873 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:41.874 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.890+0000 7f5f0592e640 5 monclient: authenticate success, global_id 4514 2024-03-29T10:58:41.874 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.890+0000 7f5f0592e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:41.874 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.890+0000 7f5efdffb640 1 -- 172.21.15.112:0/1242616531 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ef00017d0 con 0x7f5f00142800 2024-03-29T10:58:41.874 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.890+0000 7f5efdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:41.874 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.890+0000 7f5efdffb640 1 -- 172.21.15.112:0/1242616531 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f5ef0001c20 con 0x7f5f00142800 2024-03-29T10:58:41.874 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.890+0000 7f5efdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:41.874 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.890+0000 7f5efdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:41.874 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.890+0000 7f5efdffb640 10 monclient: dump: 2024-03-29T10:58:41.874 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:41.874 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:41.874 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:41.874 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:41.874 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:41.874 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:41.875 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:41.875 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:41.875 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:41.875 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:41.875 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.890+0000 7f5f0592e640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:41.875 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.890+0000 7f5f0592e640 4 set_mon_vals no callback set 2024-03-29T10:58:41.877 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.891+0000 7f5f0592e640 10 monclient: shutdown 2024-03-29T10:58:41.878 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.891+0000 7f5f0592e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:41.878 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.891+0000 7f5f0592e640 1 -- 172.21.15.112:0/1242616531 >> v1:172.21.15.112:6790/0 conn(0x7f5f00142800 legacy=0x7f5f0014da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:41.878 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.892+0000 7f5f0592e640 1 -- 172.21.15.112:0/1242616531 shutdown_connections 2024-03-29T10:58:41.878 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.892+0000 7f5f0592e640 1 -- 172.21.15.112:0/1242616531 >> 172.21.15.112:0/1242616531 conn(0x7f5f000b35e0 msgr2=0x7f5f000b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:41.878 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.892+0000 7f5f0592e640 1 -- 172.21.15.112:0/1242616531 shutdown_connections 2024-03-29T10:58:41.878 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.893+0000 7f5f0592e640 1 -- 172.21.15.112:0/1242616531 wait complete. 2024-03-29T10:58:41.878 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.893+0000 7f5f0592e640 10 monclient: build_initial_monmap 2024-03-29T10:58:41.878 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.893+0000 7f5f0592e640 10 monclient: monmap: 2024-03-29T10:58:41.878 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:58:41.878 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:41.878 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:58:41.878 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:58:41.878 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:41.879 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:41.879 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:58:41.879 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:58:41.879 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:58:41.879 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:41.879 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.894+0000 7f5f0592e640 1 librados: starting msgr at 2024-03-29T10:58:41.879 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.894+0000 7f5f0592e640 1 librados: starting objecter 2024-03-29T10:58:41.879 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.894+0000 7f5f0592e640 1 Processor -- start 2024-03-29T10:58:41.879 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.894+0000 7f5f0592e640 1 -- start start 2024-03-29T10:58:41.879 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.894+0000 7f5f0592e640 1 librados: setting wanted keys 2024-03-29T10:58:41.879 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.894+0000 7f5f0592e640 1 librados: calling monclient init 2024-03-29T10:58:41.879 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.894+0000 7f5f0592e640 10 monclient: init 2024-03-29T10:58:41.879 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.895+0000 7f5f0592e640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:41.879 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.895+0000 7f5f0592e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:58:41.879 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.895+0000 7f5f0592e640 10 monclient(hunting): picked mon.noname-c con 0x7f5f000ae250 addr v1:172.21.15.112:6790/0 2024-03-29T10:58:41.880 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.895+0000 7f5efeffd640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f5f000ae250 0x7f5f001da7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:38660/0 (socket says 172.21.15.112:38660) 2024-03-29T10:58:41.880 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.895+0000 7f5efeffd640 1 -- 172.21.15.112:0/3634948913 learned_addr learned my addr 172.21.15.112:0/3634948913 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:58:41.880 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.895+0000 7f5ee7fff640 10 client.?.objecter ms_handle_connect 0x7f5f000ae250 2024-03-29T10:58:41.880 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.895+0000 7f5ee7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:41.880 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.896+0000 7f5f0592e640 10 monclient(hunting): picked mon.noname-a con 0x7f5f0013f970 addr v1:172.21.15.112:6789/0 2024-03-29T10:58:41.880 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.896+0000 7f5ee7fff640 10 client.?.objecter ms_handle_connect 0x7f5f0013f970 2024-03-29T10:58:41.880 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.896+0000 7f5ee7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:41.882 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.897+0000 7f5f0592e640 10 monclient(hunting): picked mon.noname-b con 0x7f5f00142800 addr v1:172.21.15.118:6789/0 2024-03-29T10:58:41.882 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.897+0000 7f5f0592e640 1 -- 172.21.15.112:0/3634948913 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f001daed0 con 0x7f5f0013f970 2024-03-29T10:58:41.882 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.897+0000 7f5f0592e640 1 -- 172.21.15.112:0/3634948913 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f001db060 con 0x7f5f00142800 2024-03-29T10:58:41.882 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.897+0000 7f5f0592e640 1 -- 172.21.15.112:0/3634948913 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f00145c10 con 0x7f5f000ae250 2024-03-29T10:58:41.882 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.897+0000 7f5f0592e640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:41.882 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.897+0000 7f5f0592e640 10 monclient(hunting): authenticate will time out at 1277.202455s 2024-03-29T10:58:41.882 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.898+0000 7f5ee7fff640 1 -- 172.21.15.112:0/3634948913 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889521799 0 0) 0x7f5f001daed0 con 0x7f5f0013f970 2024-03-29T10:58:41.882 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.898+0000 7f5ee7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:41.882 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.898+0000 7f5ee7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:41.882 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.898+0000 7f5ee7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:41.883 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.898+0000 7f5ee7fff640 10 monclient(hunting): my global_id is 4488 2024-03-29T10:58:41.883 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.898+0000 7f5ee7fff640 1 -- 172.21.15.112:0/3634948913 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eec004eb0 con 0x7f5f0013f970 2024-03-29T10:58:41.883 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.898+0000 7f5ee7fff640 1 -- 172.21.15.112:0/3634948913 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1975375783 0 0) 0x7f5f00145c10 con 0x7f5f000ae250 2024-03-29T10:58:41.883 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.898+0000 7f5ee7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:41.883 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.898+0000 7f5ee7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:41.883 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.898+0000 7f5ee7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:41.883 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.898+0000 7f5ee7fff640 10 monclient(hunting): my global_id is 4517 2024-03-29T10:58:41.883 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.898+0000 7f5ee7fff640 1 -- 172.21.15.112:0/3634948913 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f001daed0 con 0x7f5f000ae250 2024-03-29T10:58:41.883 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.898+0000 7f5ee7fff640 1 -- 172.21.15.112:0/3634948913 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1830249499 0 0) 0x7f5eec004eb0 con 0x7f5f0013f970 2024-03-29T10:58:41.883 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.898+0000 7f5ee7fff640 1 -- 172.21.15.112:0/3634948913 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f00145c10 con 0x7f5f0013f970 2024-03-29T10:58:41.883 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.898+0000 7f5ee7fff640 1 -- 172.21.15.112:0/3634948913 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f5ef4001a70 con 0x7f5f0013f970 2024-03-29T10:58:41.883 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5ee7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:41.883 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5ee7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:58:41.883 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5ee7fff640 10 monclient(hunting): dump: 2024-03-29T10:58:41.884 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:41.884 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:41.884 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:41.884 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:41.884 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:41.884 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:41.884 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:41.884 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:41.884 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:41.884 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:41.884 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5ee7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:41.884 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5ee7fff640 1 -- 172.21.15.112:0/3634948913 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1112297559 0 0) 0x7f5f001daed0 con 0x7f5f000ae250 2024-03-29T10:58:41.884 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5ee7fff640 1 -- 172.21.15.112:0/3634948913 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5eec004eb0 con 0x7f5f000ae250 2024-03-29T10:58:41.884 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5ee7fff640 1 -- 172.21.15.112:0/3634948913 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f5ef8001b30 con 0x7f5f000ae250 2024-03-29T10:58:41.885 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5ee7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:41.885 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5ee7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:41.885 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5ee7fff640 10 monclient(hunting): dump: 2024-03-29T10:58:41.885 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:41.885 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:41.885 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:41.885 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:41.885 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:41.885 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:41.885 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:41.885 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:41.885 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:41.885 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:41.885 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5ee7fff640 1 -- 172.21.15.112:0/3634948913 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2850092528 0 0) 0x7f5f00145c10 con 0x7f5f0013f970 2024-03-29T10:58:41.885 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5ee7fff640 1 -- 172.21.15.112:0/3634948913 >> v1:172.21.15.112:6790/0 conn(0x7f5f000ae250 legacy=0x7f5f001da7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:41.886 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5ee7fff640 1 -- 172.21.15.112:0/3634948913 >> v1:172.21.15.118:6789/0 conn(0x7f5f00142800 legacy=0x7f5f0013ae50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:58:41.886 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5ee7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:58:41.886 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5ee7fff640 1 monclient: found mon.a 2024-03-29T10:58:41.886 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5ee7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:41.886 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5ee7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:58:41.886 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5ee7fff640 1 -- 172.21.15.112:0/3634948913 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f00145da0 con 0x7f5f0013f970 2024-03-29T10:58:41.886 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5ee7fff640 10 monclient: _finish_auth 0 2024-03-29T10:58:41.886 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5ee7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:58:41.886 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5ee7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:41.886 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5f0592e640 5 monclient: authenticate success, global_id 4488 2024-03-29T10:58:41.887 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5f0592e640 10 monclient: _renew_subs 2024-03-29T10:58:41.887 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5f0592e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:58:41.887 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5f0592e640 1 -- 172.21.15.112:0/3634948913 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f001e2aa0 con 0x7f5f0013f970 2024-03-29T10:58:41.887 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5f0592e640 10 client.4488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:41.887 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5f0592e640 10 monclient: _renew_subs 2024-03-29T10:58:41.887 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5f0592e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:58:41.887 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5f0592e640 1 -- 172.21.15.112:0/3634948913 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f001e2e10 con 0x7f5f0013f970 2024-03-29T10:58:41.887 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.899+0000 7f5f0592e640 1 librados: init done 2024-03-29T10:58:41.887 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.900+0000 7f5ee7fff640 1 -- 172.21.15.112:0/3634948913 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ef40047e0 con 0x7f5f0013f970 2024-03-29T10:58:41.887 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.900+0000 7f5ee7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:41.888 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.900+0000 7f5ee7fff640 1 -- 172.21.15.112:0/3634948913 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f5ef4004af0 con 0x7f5f0013f970 2024-03-29T10:58:41.888 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.900+0000 7f5ee7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:41.888 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.900+0000 7f5ee7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:41.888 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.900+0000 7f5ee7fff640 10 monclient: dump: 2024-03-29T10:58:41.888 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:41.888 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:41.888 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:41.888 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:41.888 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:41.888 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:41.888 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:41.888 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:41.888 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:41.888 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:41.889 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.900+0000 7f5efd7fa640 4 set_mon_vals no callback set 2024-03-29T10:58:41.889 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.900+0000 7f5f0592e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:41.889 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.902+0000 7f5ee7fff640 1 -- 172.21.15.112:0/3634948913 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f5ef401aac0 con 0x7f5f0013f970 2024-03-29T10:58:41.889 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.902+0000 7f5f0592e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:41.889 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.902+0000 7f5f0592e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:58:41.889 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.902+0000 7f5f0592e640 1 -- 172.21.15.112:0/3634948913 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f000acf20 con 0x7f5f0013f970 2024-03-29T10:58:41.890 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.902+0000 7f5ee7fff640 1 -- 172.21.15.112:0/3634948913 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f5ef4084ad0 con 0x7f5f0013f970 2024-03-29T10:58:41.890 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.902+0000 7f5ee7fff640 10 client.4488.objecter ms_dispatch 0x7f5f00000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:41.890 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.902+0000 7f5ee7fff640 3 client.4488.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:41.890 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.902+0000 7f5ee7fff640 3 client.4488.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:41.890 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.902+0000 7f5efeffd640 10 monclient: get_auth_request con 0x7f5eec06de90 auth_method 0 2024-03-29T10:58:41.890 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.903+0000 7f5ee7fff640 20 client.4488.objecter dump_active .. 0 homeless 2024-03-29T10:58:41.890 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.903+0000 7f5ee7fff640 10 client.4488.objecter ms_handle_connect 0x7f5eec06de90 2024-03-29T10:58:41.892 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.910+0000 7f5ee7fff640 1 -- 172.21.15.112:0/3634948913 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f000acf20 con 0x7f5f0013f970 2024-03-29T10:58:41.892 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.910+0000 7f5ee7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:41.892 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:41.910+0000 7f5ee7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:42.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:42.223+0000 7f5f0592e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:42.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:42.223+0000 7f5f0592e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:42.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:42.223+0000 7f5f0592e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:58:42.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:42.223+0000 7f5f0592e640 1 -- 172.21.15.112:0/3634948913 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f0013a460 con 0x7f5f0013f970 2024-03-29T10:58:42.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:42.224+0000 7f5ee7fff640 1 -- 172.21.15.112:0/3634948913 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f5f0013a460 con 0x7f5f0013f970 2024-03-29T10:58:42.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:42.224+0000 7f5ee7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:42.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:42.224+0000 7f5ee7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:42.206 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:58:42.206 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:42.210 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:42.228+0000 7f5f0592e640 10 librados: watch_flush enter 2024-03-29T10:58:42.210 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:42.228+0000 7f5f0592e640 10 librados: watch_flush exit 2024-03-29T10:58:42.210 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:42.228+0000 7f5f0592e640 20 client.4488.objecter shutdown clearing up homeless session... 2024-03-29T10:58:42.210 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:42.228+0000 7f5f0592e640 10 client.4488.objecter successfully canceled tick 2024-03-29T10:58:42.210 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:42.228+0000 7f5f0592e640 1 -- 172.21.15.112:0/3634948913 >> v1:172.21.15.118:6816/3181073408 conn(0x7f5eec06de90 legacy=0x7f5eec070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:42.210 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:42.228+0000 7f5f0592e640 10 monclient: shutdown 2024-03-29T10:58:42.210 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:42.228+0000 7f5f0592e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:42.210 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:42.228+0000 7f5f0592e640 1 -- 172.21.15.112:0/3634948913 >> v1:172.21.15.112:6789/0 conn(0x7f5f0013f970 legacy=0x7f5f0013a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:42.210 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:42.228+0000 7f5f0592e640 1 -- 172.21.15.112:0/3634948913 shutdown_connections 2024-03-29T10:58:42.211 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:42.228+0000 7f5f0592e640 1 -- 172.21.15.112:0/3634948913 >> 172.21.15.112:0/3634948913 conn(0x7f5f000b35e0 msgr2=0x7f5f000b2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:42.211 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:42.229+0000 7f5f0592e640 1 -- 172.21.15.112:0/3634948913 shutdown_connections 2024-03-29T10:58:42.211 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:42.229+0000 7f5f0592e640 1 -- 172.21.15.112:0/3634948913 wait complete. 2024-03-29T10:58:42.211 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:42.229+0000 7f5f0592e640 1 librados: shutdown 2024-03-29T10:58:42.224 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:49.031 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:49.196 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.216+0000 7f5ed82e2640 10 monclient: get_monmap_and_config 2024-03-29T10:58:49.197 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.216+0000 7f5ed82e2640 10 monclient: build_initial_monmap 2024-03-29T10:58:49.197 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.216+0000 7f5ed82e2640 10 monclient: monmap: 2024-03-29T10:58:49.197 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:58:49.197 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:49.197 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:58:49.217132+0000 2024-03-29T10:58:49.197 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:58:49.217132+0000 2024-03-29T10:58:49.197 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:49.197 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:49.197 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:58:49.197 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:58:49.197 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:58:49.197 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:49.198 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.217+0000 7f5ed82e2640 1 Processor -- start 2024-03-29T10:58:49.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.217+0000 7f5ed82e2640 1 -- start start 2024-03-29T10:58:49.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.217+0000 7f5ed82e2640 10 monclient: init 2024-03-29T10:58:49.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.218+0000 7f5ed82e2640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:49.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.218+0000 7f5ed82e2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:58:49.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.218+0000 7f5ed82e2640 10 monclient(hunting): picked mon.noname-a con 0x7f5ed00b0770 addr v1:172.21.15.112:6789/0 2024-03-29T10:58:49.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.218+0000 7f5ed82e2640 10 monclient(hunting): picked mon.noname-c con 0x7f5ed014a700 addr v1:172.21.15.118:6789/0 2024-03-29T10:58:49.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.218+0000 7f5ed82e2640 10 monclient(hunting): picked mon.noname-b con 0x7f5ed00ac5a0 addr v1:172.21.15.112:6790/0 2024-03-29T10:58:49.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.218+0000 7f5ed82e2640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed00aba70 con 0x7f5ed00b0770 2024-03-29T10:58:49.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.218+0000 7f5ed82e2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed00aecb0 con 0x7f5ed014a700 2024-03-29T10:58:49.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.218+0000 7f5ed82e2640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed00af150 con 0x7f5ed00ac5a0 2024-03-29T10:58:49.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.218+0000 7f5ed82e2640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:49.199 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.218+0000 7f5ed82e2640 10 monclient(hunting): authenticate will time out at 1284.522722s 2024-03-29T10:58:49.200 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.218+0000 7f5ed687f640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f5ed00ac5a0 0x7f5ed0151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:33284/0 (socket says 172.21.15.112:33284) 2024-03-29T10:58:49.200 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.218+0000 7f5ed687f640 1 -- 172.21.15.112:0/527416460 learned_addr learned my addr 172.21.15.112:0/527416460 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:58:49.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.219+0000 7f5ed507c640 1 -- 172.21.15.112:0/527416460 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1217795284 0 0) 0x7f5ed00af150 con 0x7f5ed00ac5a0 2024-03-29T10:58:49.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.219+0000 7f5ed507c640 10 monclient(hunting): _negotiate 2024-03-29T10:58:49.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.219+0000 7f5ed507c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:49.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.219+0000 7f5ed507c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:49.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.219+0000 7f5ed507c640 10 monclient(hunting): my global_id is 4520 2024-03-29T10:58:49.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.219+0000 7f5ed507c640 1 -- 172.21.15.112:0/527416460 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ebc003520 con 0x7f5ed00ac5a0 2024-03-29T10:58:49.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.219+0000 7f5ed507c640 1 -- 172.21.15.112:0/527416460 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2607355430 0 0) 0x7f5ed00aba70 con 0x7f5ed00b0770 2024-03-29T10:58:49.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.219+0000 7f5ed507c640 10 monclient(hunting): _negotiate 2024-03-29T10:58:49.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.219+0000 7f5ed507c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:49.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.219+0000 7f5ed507c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:49.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.219+0000 7f5ed507c640 10 monclient(hunting): my global_id is 4491 2024-03-29T10:58:49.202 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.219+0000 7f5ed507c640 1 -- 172.21.15.112:0/527416460 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ed00af150 con 0x7f5ed00b0770 2024-03-29T10:58:49.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.220+0000 7f5ed507c640 1 -- 172.21.15.112:0/527416460 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950909478 0 0) 0x7f5ed00aecb0 con 0x7f5ed014a700 2024-03-29T10:58:49.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.220+0000 7f5ed507c640 10 monclient(hunting): _negotiate 2024-03-29T10:58:49.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.220+0000 7f5ed507c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:49.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.220+0000 7f5ed507c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:49.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.220+0000 7f5ed507c640 10 monclient(hunting): my global_id is 4477 2024-03-29T10:58:49.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.220+0000 7f5ed507c640 1 -- 172.21.15.112:0/527416460 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ed00aba70 con 0x7f5ed014a700 2024-03-29T10:58:49.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.220+0000 7f5ed507c640 1 -- 172.21.15.112:0/527416460 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1054681470 0 0) 0x7f5ebc003520 con 0x7f5ed00ac5a0 2024-03-29T10:58:49.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.220+0000 7f5ed507c640 1 -- 172.21.15.112:0/527416460 >> v1:172.21.15.118:6789/0 conn(0x7f5ed014a700 legacy=0x7f5ed00abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:49.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.220+0000 7f5ed507c640 1 -- 172.21.15.112:0/527416460 >> v1:172.21.15.112:6789/0 conn(0x7f5ed00b0770 legacy=0x7f5ed0149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:49.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.220+0000 7f5ed507c640 10 monclient: _finish_hunting 0 2024-03-29T10:58:49.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.220+0000 7f5ed507c640 1 monclient: found mon.noname-b 2024-03-29T10:58:49.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.220+0000 7f5ed507c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:49.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.220+0000 7f5ed507c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:58:49.203 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.220+0000 7f5ed507c640 1 -- 172.21.15.112:0/527416460 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed00ac9d0 con 0x7f5ed00ac5a0 2024-03-29T10:58:49.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.220+0000 7f5ed507c640 10 monclient: _finish_auth 0 2024-03-29T10:58:49.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.220+0000 7f5ed507c640 10 monclient: _check_auth_tickets 2024-03-29T10:58:49.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.220+0000 7f5ed507c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:49.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.220+0000 7f5ed82e2640 5 monclient: authenticate success, global_id 4520 2024-03-29T10:58:49.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.220+0000 7f5ed82e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:49.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.220+0000 7f5ed507c640 1 -- 172.21.15.112:0/527416460 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f5ecc001690 con 0x7f5ed00ac5a0 2024-03-29T10:58:49.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.220+0000 7f5ed507c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:49.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.220+0000 7f5ed507c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:49.204 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.220+0000 7f5ed507c640 10 monclient: dump: 2024-03-29T10:58:49.204 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:49.205 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:49.205 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:49.205 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:49.205 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:49.205 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:49.205 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:49.205 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:49.205 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:49.205 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:49.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.220+0000 7f5ed82e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:49.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.222+0000 7f5ed507c640 1 -- 172.21.15.112:0/527416460 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ecc0018b0 con 0x7f5ed00ac5a0 2024-03-29T10:58:49.205 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.222+0000 7f5ed507c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:49.210 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.222+0000 7f5ed82e2640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:49.211 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.222+0000 7f5ed82e2640 4 set_mon_vals no callback set 2024-03-29T10:58:49.211 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.222+0000 7f5ed507c640 1 -- 172.21.15.112:0/527416460 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f5ecc001c40 con 0x7f5ed00ac5a0 2024-03-29T10:58:49.211 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.222+0000 7f5ed507c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:49.211 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.222+0000 7f5ed507c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:49.211 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.222+0000 7f5ed507c640 10 monclient: dump: 2024-03-29T10:58:49.211 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:49.211 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:49.211 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:49.211 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:49.211 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:49.211 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:49.211 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:49.211 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:49.211 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:49.212 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:49.212 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.222+0000 7f5ed82e2640 10 monclient: shutdown 2024-03-29T10:58:49.212 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.222+0000 7f5ed82e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:49.212 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.222+0000 7f5ed82e2640 1 -- 172.21.15.112:0/527416460 >> v1:172.21.15.112:6790/0 conn(0x7f5ed00ac5a0 legacy=0x7f5ed0151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:49.212 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.223+0000 7f5ed82e2640 1 -- 172.21.15.112:0/527416460 shutdown_connections 2024-03-29T10:58:49.212 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.223+0000 7f5ed82e2640 1 -- 172.21.15.112:0/527416460 >> 172.21.15.112:0/527416460 conn(0x7f5ed006b660 msgr2=0x7f5ed006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:49.212 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.223+0000 7f5ed82e2640 1 -- 172.21.15.112:0/527416460 shutdown_connections 2024-03-29T10:58:49.212 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.223+0000 7f5ed82e2640 1 -- 172.21.15.112:0/527416460 wait complete. 2024-03-29T10:58:49.212 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.224+0000 7f5ed82e2640 10 monclient: build_initial_monmap 2024-03-29T10:58:49.212 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.224+0000 7f5ed82e2640 10 monclient: monmap: 2024-03-29T10:58:49.212 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:58:49.212 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:49.212 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:58:49.212 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:58:49.212 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:49.213 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:49.213 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:58:49.213 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:58:49.213 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:58:49.213 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:49.213 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.224+0000 7f5ed82e2640 1 librados: starting msgr at 2024-03-29T10:58:49.213 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.224+0000 7f5ed82e2640 1 librados: starting objecter 2024-03-29T10:58:49.213 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.224+0000 7f5ed82e2640 1 Processor -- start 2024-03-29T10:58:49.213 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.224+0000 7f5ed82e2640 1 -- start start 2024-03-29T10:58:49.213 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.224+0000 7f5ed82e2640 1 librados: setting wanted keys 2024-03-29T10:58:49.213 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.224+0000 7f5ed82e2640 1 librados: calling monclient init 2024-03-29T10:58:49.213 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.224+0000 7f5ed82e2640 10 monclient: init 2024-03-29T10:58:49.213 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.225+0000 7f5ed82e2640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:49.213 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.225+0000 7f5ed82e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:58:49.213 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.225+0000 7f5ed82e2640 10 monclient(hunting): picked mon.noname-a con 0x7f5ed00ac5a0 addr v1:172.21.15.112:6789/0 2024-03-29T10:58:49.214 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.225+0000 7f5ed82e2640 10 monclient(hunting): picked mon.noname-c con 0x7f5ed00b0770 addr v1:172.21.15.112:6790/0 2024-03-29T10:58:49.214 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.225+0000 7f5ed82e2640 10 monclient(hunting): picked mon.noname-b con 0x7f5ed014a700 addr v1:172.21.15.118:6789/0 2024-03-29T10:58:49.214 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.225+0000 7f5ed82e2640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed01239d0 con 0x7f5ed00ac5a0 2024-03-29T10:58:49.214 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.225+0000 7f5ed82e2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed01273c0 con 0x7f5ed014a700 2024-03-29T10:58:49.214 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.225+0000 7f5ed82e2640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed0128120 con 0x7f5ed00b0770 2024-03-29T10:58:49.214 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.225+0000 7f5ed82e2640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:49.214 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.225+0000 7f5ed82e2640 10 monclient(hunting): authenticate will time out at 1284.530092s 2024-03-29T10:58:49.214 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.226+0000 7f5ed587d640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f5ed00b0770 0x7f5ed0144fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:33510/0 (socket says 172.21.15.112:33510) 2024-03-29T10:58:49.214 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.226+0000 7f5ed587d640 1 -- 172.21.15.112:0/1311660138 learned_addr learned my addr 172.21.15.112:0/1311660138 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:58:49.214 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.226+0000 7f5ec6ffd640 10 client.?.objecter ms_handle_connect 0x7f5ed014a700 2024-03-29T10:58:49.214 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.226+0000 7f5ec6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:49.214 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.226+0000 7f5ec6ffd640 10 client.?.objecter ms_handle_connect 0x7f5ed00b0770 2024-03-29T10:58:49.214 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.226+0000 7f5ec6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:49.215 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.226+0000 7f5ec6ffd640 1 -- 172.21.15.112:0/1311660138 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3593149611 0 0) 0x7f5ed01273c0 con 0x7f5ed014a700 2024-03-29T10:58:49.215 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.226+0000 7f5ec6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:49.215 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.226+0000 7f5ec6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:49.215 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.226+0000 7f5ec6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:49.215 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.226+0000 7f5ec6ffd640 10 monclient(hunting): my global_id is 4480 2024-03-29T10:58:49.215 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.226+0000 7f5ec6ffd640 1 -- 172.21.15.112:0/1311660138 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb4004eb0 con 0x7f5ed014a700 2024-03-29T10:58:49.215 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.227+0000 7f5ec6ffd640 1 -- 172.21.15.112:0/1311660138 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954258412 0 0) 0x7f5ed0128120 con 0x7f5ed00b0770 2024-03-29T10:58:49.215 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.227+0000 7f5ec6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:49.215 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.227+0000 7f5ec6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:49.215 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.227+0000 7f5ec6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:49.215 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.227+0000 7f5ec6ffd640 10 monclient(hunting): my global_id is 4523 2024-03-29T10:58:49.215 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.227+0000 7f5ec6ffd640 1 -- 172.21.15.112:0/1311660138 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ed01273c0 con 0x7f5ed00b0770 2024-03-29T10:58:49.215 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.227+0000 7f5ec6ffd640 10 client.?.objecter ms_handle_connect 0x7f5ed00ac5a0 2024-03-29T10:58:49.215 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.227+0000 7f5ec6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:49.216 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.227+0000 7f5ec6ffd640 1 -- 172.21.15.112:0/1311660138 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1854166599 0 0) 0x7f5eb4004eb0 con 0x7f5ed014a700 2024-03-29T10:58:49.216 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.227+0000 7f5ec6ffd640 1 -- 172.21.15.112:0/1311660138 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ed0128120 con 0x7f5ed014a700 2024-03-29T10:58:49.216 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.227+0000 7f5ec6ffd640 1 -- 172.21.15.112:0/1311660138 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f5ecc0015d0 con 0x7f5ed014a700 2024-03-29T10:58:49.216 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.227+0000 7f5ec6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:49.216 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.227+0000 7f5ec6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:49.216 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.227+0000 7f5ec6ffd640 10 monclient(hunting): dump: 2024-03-29T10:58:49.216 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:49.216 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:49.216 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:49.216 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:49.216 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:49.216 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:49.216 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:49.216 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:49.216 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:49.217 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:49.217 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.227+0000 7f5ec6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:49.217 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ec6ffd640 1 -- 172.21.15.112:0/1311660138 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 220091334 0 0) 0x7f5ed01273c0 con 0x7f5ed00b0770 2024-03-29T10:58:49.217 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ec6ffd640 1 -- 172.21.15.112:0/1311660138 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5eb4004eb0 con 0x7f5ed00b0770 2024-03-29T10:58:49.217 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ec6ffd640 1 -- 172.21.15.112:0/1311660138 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f5ec00018c0 con 0x7f5ed00b0770 2024-03-29T10:58:49.217 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ec6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:49.217 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ec6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:49.217 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ec6ffd640 10 monclient(hunting): dump: 2024-03-29T10:58:49.217 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:49.217 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:49.217 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:49.217 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:49.217 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:49.218 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:49.218 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:49.218 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:49.218 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:49.218 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:49.218 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ec6ffd640 1 -- 172.21.15.112:0/1311660138 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1915541383 0 0) 0x7f5ed0128120 con 0x7f5ed014a700 2024-03-29T10:58:49.218 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ec6ffd640 1 -- 172.21.15.112:0/1311660138 >> v1:172.21.15.112:6790/0 conn(0x7f5ed00b0770 legacy=0x7f5ed0144fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:49.218 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ec6ffd640 1 -- 172.21.15.112:0/1311660138 >> v1:172.21.15.112:6789/0 conn(0x7f5ed00ac5a0 legacy=0x7f5ed0126bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:49.218 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ec6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:58:49.218 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ec6ffd640 1 monclient: found mon.b 2024-03-29T10:58:49.218 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ec6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:49.218 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ec6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:58:49.218 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ec6ffd640 1 -- 172.21.15.112:0/1311660138 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed0128400 con 0x7f5ed014a700 2024-03-29T10:58:49.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ec6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:58:49.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ec6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:58:49.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ec6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:49.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ed82e2640 5 monclient: authenticate success, global_id 4480 2024-03-29T10:58:49.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ed82e2640 10 monclient: _renew_subs 2024-03-29T10:58:49.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ed82e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:58:49.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ed82e2640 1 -- 172.21.15.112:0/1311660138 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed0125fa0 con 0x7f5ed014a700 2024-03-29T10:58:49.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ed82e2640 10 client.4480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:49.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ed82e2640 10 monclient: _renew_subs 2024-03-29T10:58:49.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ed82e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:58:49.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ed82e2640 1 -- 172.21.15.112:0/1311660138 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed01456b0 con 0x7f5ed014a700 2024-03-29T10:58:49.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ed82e2640 1 librados: init done 2024-03-29T10:58:49.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ec6ffd640 1 -- 172.21.15.112:0/1311660138 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ecc001730 con 0x7f5ed014a700 2024-03-29T10:58:49.219 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ec6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:49.220 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ec6ffd640 1 -- 172.21.15.112:0/1311660138 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f5ecc001220 con 0x7f5ed014a700 2024-03-29T10:58:49.220 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ec6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:49.220 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ec6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:49.220 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ed487b640 4 set_mon_vals no callback set 2024-03-29T10:58:49.220 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.228+0000 7f5ec6ffd640 10 monclient: dump: 2024-03-29T10:58:49.220 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:49.220 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:49.220 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:49.220 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:49.220 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:49.220 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:49.220 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:49.220 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:49.220 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:49.220 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:49.221 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.229+0000 7f5ed82e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:49.221 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.230+0000 7f5ed82e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:49.221 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.230+0000 7f5ed82e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:58:49.221 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.230+0000 7f5ed82e2640 1 -- 172.21.15.112:0/1311660138 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb8005080 con 0x7f5ed014a700 2024-03-29T10:58:49.221 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.231+0000 7f5ec6ffd640 1 -- 172.21.15.112:0/1311660138 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f5ecc01aaa0 con 0x7f5ed014a700 2024-03-29T10:58:49.222 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.231+0000 7f5ed607e640 10 monclient: get_auth_request con 0x7f5eb406dfc0 auth_method 0 2024-03-29T10:58:49.222 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.231+0000 7f5ec6ffd640 1 -- 172.21.15.112:0/1311660138 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f5ecc0849b0 con 0x7f5ed014a700 2024-03-29T10:58:49.222 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.232+0000 7f5ec6ffd640 10 client.4480.objecter ms_dispatch 0x7f5ed0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:49.222 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.232+0000 7f5ec6ffd640 3 client.4480.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:49.222 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.232+0000 7f5ec6ffd640 3 client.4480.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:49.222 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.232+0000 7f5ec6ffd640 20 client.4480.objecter dump_active .. 0 homeless 2024-03-29T10:58:49.222 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.232+0000 7f5ec6ffd640 10 client.4480.objecter ms_handle_connect 0x7f5eb406dfc0 2024-03-29T10:58:49.222 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.238+0000 7f5ec6ffd640 1 -- 172.21.15.112:0/1311660138 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ecc0500c0 con 0x7f5ed014a700 2024-03-29T10:58:49.222 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.238+0000 7f5ec6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:49.222 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.238+0000 7f5ec6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:49.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.616+0000 7f5ed82e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:49.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.616+0000 7f5ed82e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:49.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.616+0000 7f5ed82e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:58:49.597 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.616+0000 7f5ed82e2640 1 -- 172.21.15.112:0/1311660138 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5eb8005330 con 0x7f5ed014a700 2024-03-29T10:58:49.598 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.617+0000 7f5ec6ffd640 1 -- 172.21.15.112:0/1311660138 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f5ecc0521c0 con 0x7f5ed014a700 2024-03-29T10:58:49.598 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.617+0000 7f5ec6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:49.598 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.617+0000 7f5ec6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:49.598 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:58:49.598 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:49.602 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.621+0000 7f5ed82e2640 10 librados: watch_flush enter 2024-03-29T10:58:49.602 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.621+0000 7f5ed82e2640 10 librados: watch_flush exit 2024-03-29T10:58:49.602 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.621+0000 7f5ed82e2640 20 client.4480.objecter shutdown clearing up homeless session... 2024-03-29T10:58:49.602 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.621+0000 7f5ed82e2640 10 client.4480.objecter successfully canceled tick 2024-03-29T10:58:49.602 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.621+0000 7f5ed82e2640 1 -- 172.21.15.112:0/1311660138 >> v1:172.21.15.118:6816/3181073408 conn(0x7f5eb406dfc0 legacy=0x7f5eb4070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:49.602 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.621+0000 7f5ed82e2640 10 monclient: shutdown 2024-03-29T10:58:49.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.621+0000 7f5ed82e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:49.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.621+0000 7f5ed82e2640 1 -- 172.21.15.112:0/1311660138 >> v1:172.21.15.118:6789/0 conn(0x7f5ed014a700 legacy=0x7f5ed01db890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:49.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.622+0000 7f5ed82e2640 1 -- 172.21.15.112:0/1311660138 shutdown_connections 2024-03-29T10:58:49.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.622+0000 7f5ed82e2640 1 -- 172.21.15.112:0/1311660138 >> 172.21.15.112:0/1311660138 conn(0x7f5ed006b660 msgr2=0x7f5ed006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:49.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.622+0000 7f5ed82e2640 1 -- 172.21.15.112:0/1311660138 shutdown_connections 2024-03-29T10:58:49.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.622+0000 7f5ed82e2640 1 -- 172.21.15.112:0/1311660138 wait complete. 2024-03-29T10:58:49.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:49.622+0000 7f5ed82e2640 1 librados: shutdown 2024-03-29T10:58:49.618 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:56.424 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:56.566 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.585+0000 7f1d43fc6640 10 monclient: get_monmap_and_config 2024-03-29T10:58:56.566 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.585+0000 7f1d43fc6640 10 monclient: build_initial_monmap 2024-03-29T10:58:56.566 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.585+0000 7f1d43fc6640 10 monclient: monmap: 2024-03-29T10:58:56.566 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:58:56.566 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:56.566 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:58:56.586734+0000 2024-03-29T10:58:56.566 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:58:56.586734+0000 2024-03-29T10:58:56.566 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:56.567 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:56.567 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:58:56.567 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:58:56.567 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:58:56.567 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:56.567 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.586+0000 7f1d43fc6640 1 Processor -- start 2024-03-29T10:58:56.568 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.586+0000 7f1d43fc6640 1 -- start start 2024-03-29T10:58:56.568 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.586+0000 7f1d43fc6640 10 monclient: init 2024-03-29T10:58:56.568 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.587+0000 7f1d43fc6640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:56.568 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.587+0000 7f1d43fc6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:58:56.568 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.587+0000 7f1d43fc6640 10 monclient(hunting): picked mon.noname-c con 0x7f1d3c1457f0 addr v1:172.21.15.118:6789/0 2024-03-29T10:58:56.570 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.587+0000 7f1d43fc6640 10 monclient(hunting): picked mon.noname-b con 0x7f1d3c13f650 addr v1:172.21.15.112:6790/0 2024-03-29T10:58:56.570 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.587+0000 7f1d43fc6640 10 monclient(hunting): picked mon.noname-a con 0x7f1d3c0a9d70 addr v1:172.21.15.112:6789/0 2024-03-29T10:58:56.571 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.587+0000 7f1d43fc6640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3c0aaef0 con 0x7f1d3c0a9d70 2024-03-29T10:58:56.571 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.587+0000 7f1d43fc6640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3c145bd0 con 0x7f1d3c1457f0 2024-03-29T10:58:56.571 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.587+0000 7f1d43fc6640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3c145eb0 con 0x7f1d3c13f650 2024-03-29T10:58:56.571 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.587+0000 7f1d43fc6640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:56.571 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.587+0000 7f1d43fc6640 10 monclient(hunting): authenticate will time out at 1291.892039s 2024-03-29T10:58:56.571 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.589+0000 7f1d42563640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f1d3c0a9d70 0x7f1d3c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:53732/0 (socket says 172.21.15.112:53732) 2024-03-29T10:58:56.571 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.589+0000 7f1d42563640 1 -- 172.21.15.112:0/2569195682 learned_addr learned my addr 172.21.15.112:0/2569195682 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:58:56.571 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.589+0000 7f1d40d60640 1 -- 172.21.15.112:0/2569195682 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158177449 0 0) 0x7f1d3c0aaef0 con 0x7f1d3c0a9d70 2024-03-29T10:58:56.571 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.589+0000 7f1d40d60640 10 monclient(hunting): _negotiate 2024-03-29T10:58:56.571 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.589+0000 7f1d40d60640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:56.571 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.589+0000 7f1d40d60640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:56.571 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.589+0000 7f1d40d60640 10 monclient(hunting): my global_id is 4497 2024-03-29T10:58:56.571 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.589+0000 7f1d40d60640 1 -- 172.21.15.112:0/2569195682 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d28003520 con 0x7f1d3c0a9d70 2024-03-29T10:58:56.571 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.589+0000 7f1d40d60640 1 -- 172.21.15.112:0/2569195682 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2673208062 0 0) 0x7f1d28003520 con 0x7f1d3c0a9d70 2024-03-29T10:58:56.572 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.589+0000 7f1d40d60640 1 -- 172.21.15.112:0/2569195682 >> v1:172.21.15.112:6790/0 conn(0x7f1d3c13f650 legacy=0x7f1d3c0b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:58:56.573 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.591+0000 7f1d40d60640 1 -- 172.21.15.112:0/2569195682 >> v1:172.21.15.118:6789/0 conn(0x7f1d3c1457f0 legacy=0x7f1d3c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:58:56.574 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.591+0000 7f1d40d60640 10 monclient: _finish_hunting 0 2024-03-29T10:58:56.574 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.591+0000 7f1d40d60640 1 monclient: found mon.noname-a 2024-03-29T10:58:56.574 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.591+0000 7f1d40d60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:56.574 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.591+0000 7f1d40d60640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T10:58:56.574 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.591+0000 7f1d40d60640 1 -- 172.21.15.112:0/2569195682 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3c146340 con 0x7f1d3c0a9d70 2024-03-29T10:58:56.574 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.591+0000 7f1d40d60640 10 monclient: _finish_auth 0 2024-03-29T10:58:56.574 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.591+0000 7f1d40d60640 10 monclient: _check_auth_tickets 2024-03-29T10:58:56.574 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.591+0000 7f1d40d60640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:56.574 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.591+0000 7f1d40d60640 1 -- 172.21.15.112:0/2569195682 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1d34001a10 con 0x7f1d3c0a9d70 2024-03-29T10:58:56.574 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.591+0000 7f1d40d60640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:56.574 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.591+0000 7f1d40d60640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:58:56.574 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.591+0000 7f1d40d60640 10 monclient: dump: 2024-03-29T10:58:56.574 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:56.575 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:56.575 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:56.575 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:56.575 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:56.575 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:56.575 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:56.575 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:56.575 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:56.575 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:56.575 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.591+0000 7f1d40d60640 1 -- 172.21.15.112:0/2569195682 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d34001c70 con 0x7f1d3c0a9d70 2024-03-29T10:58:56.575 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.591+0000 7f1d40d60640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:56.575 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.591+0000 7f1d40d60640 1 -- 172.21.15.112:0/2569195682 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1d340020c0 con 0x7f1d3c0a9d70 2024-03-29T10:58:56.575 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.591+0000 7f1d40d60640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:56.575 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.591+0000 7f1d40d60640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:56.575 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.591+0000 7f1d40d60640 10 monclient: dump: 2024-03-29T10:58:56.576 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:56.576 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:56.576 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:56.576 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:56.576 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:56.576 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:56.576 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:56.576 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:56.576 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:56.576 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:56.577 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.594+0000 7f1d43fc6640 5 monclient: authenticate success, global_id 4497 2024-03-29T10:58:56.577 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.594+0000 7f1d43fc6640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:56.577 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.594+0000 7f1d43fc6640 4 set_mon_vals no callback set 2024-03-29T10:58:56.577 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.594+0000 7f1d43fc6640 10 monclient: shutdown 2024-03-29T10:58:56.577 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.594+0000 7f1d43fc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:56.577 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.594+0000 7f1d43fc6640 1 -- 172.21.15.112:0/2569195682 >> v1:172.21.15.112:6789/0 conn(0x7f1d3c0a9d70 legacy=0x7f1d3c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:56.577 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.595+0000 7f1d43fc6640 1 -- 172.21.15.112:0/2569195682 shutdown_connections 2024-03-29T10:58:56.577 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.595+0000 7f1d43fc6640 1 -- 172.21.15.112:0/2569195682 >> 172.21.15.112:0/2569195682 conn(0x7f1d3c066a00 msgr2=0x7f1d3c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:56.578 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.596+0000 7f1d43fc6640 1 -- 172.21.15.112:0/2569195682 shutdown_connections 2024-03-29T10:58:56.578 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.596+0000 7f1d43fc6640 1 -- 172.21.15.112:0/2569195682 wait complete. 2024-03-29T10:58:56.578 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.597+0000 7f1d43fc6640 10 monclient: build_initial_monmap 2024-03-29T10:58:56.578 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.597+0000 7f1d43fc6640 10 monclient: monmap: 2024-03-29T10:58:56.578 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:58:56.578 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:56.578 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:58:56.578 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:58:56.578 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:56.578 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:56.578 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:58:56.578 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:58:56.579 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:58:56.579 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:56.579 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.597+0000 7f1d43fc6640 1 librados: starting msgr at 2024-03-29T10:58:56.579 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.597+0000 7f1d43fc6640 1 librados: starting objecter 2024-03-29T10:58:56.580 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.597+0000 7f1d43fc6640 1 Processor -- start 2024-03-29T10:58:56.580 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.598+0000 7f1d43fc6640 1 -- start start 2024-03-29T10:58:56.580 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.598+0000 7f1d43fc6640 1 librados: setting wanted keys 2024-03-29T10:58:56.580 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.598+0000 7f1d43fc6640 1 librados: calling monclient init 2024-03-29T10:58:56.580 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.598+0000 7f1d43fc6640 10 monclient: init 2024-03-29T10:58:56.580 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.598+0000 7f1d43fc6640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:56.580 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.598+0000 7f1d43fc6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:58:56.580 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.598+0000 7f1d43fc6640 10 monclient(hunting): picked mon.noname-c con 0x7f1d3c0a9d70 addr v1:172.21.15.112:6790/0 2024-03-29T10:58:56.580 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.598+0000 7f1d43fc6640 10 monclient(hunting): picked mon.noname-a con 0x7f1d3c13f650 addr v1:172.21.15.112:6789/0 2024-03-29T10:58:56.583 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.601+0000 7f1d43fc6640 10 monclient(hunting): picked mon.noname-b con 0x7f1d3c1457f0 addr v1:172.21.15.118:6789/0 2024-03-29T10:58:56.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.601+0000 7f1d43fc6640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3c071110 con 0x7f1d3c13f650 2024-03-29T10:58:56.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.601+0000 7f1d43fc6640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3c138100 con 0x7f1d3c1457f0 2024-03-29T10:58:56.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.601+0000 7f1d43fc6640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3c1383e0 con 0x7f1d3c0a9d70 2024-03-29T10:58:56.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.601+0000 7f1d43fc6640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:56.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.601+0000 7f1d43fc6640 10 monclient(hunting): authenticate will time out at 1291.905901s 2024-03-29T10:58:56.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.601+0000 7f1d42563640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f1d3c1457f0 0x7f1d3c1e6040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:55035/0 (socket says 172.21.15.112:55035) 2024-03-29T10:58:56.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.601+0000 7f1d42563640 1 -- 172.21.15.112:0/1697501803 learned_addr learned my addr 172.21.15.112:0/1697501803 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:58:56.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.601+0000 7f1d327fc640 10 client.?.objecter ms_handle_connect 0x7f1d3c1457f0 2024-03-29T10:58:56.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.602+0000 7f1d327fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:56.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.602+0000 7f1d327fc640 1 -- 172.21.15.112:0/1697501803 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 470766589 0 0) 0x7f1d3c138100 con 0x7f1d3c1457f0 2024-03-29T10:58:56.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.602+0000 7f1d327fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:56.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.602+0000 7f1d327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:56.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.602+0000 7f1d327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:56.584 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.602+0000 7f1d327fc640 10 monclient(hunting): my global_id is 4483 2024-03-29T10:58:56.585 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.602+0000 7f1d327fc640 1 -- 172.21.15.112:0/1697501803 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d20004eb0 con 0x7f1d3c1457f0 2024-03-29T10:58:56.585 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.602+0000 7f1d327fc640 1 -- 172.21.15.112:0/1697501803 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3283977195 0 0) 0x7f1d20004eb0 con 0x7f1d3c1457f0 2024-03-29T10:58:56.585 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.602+0000 7f1d327fc640 1 -- 172.21.15.112:0/1697501803 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d3c138100 con 0x7f1d3c1457f0 2024-03-29T10:58:56.585 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.602+0000 7f1d327fc640 10 client.?.objecter ms_handle_connect 0x7f1d3c13f650 2024-03-29T10:58:56.585 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.602+0000 7f1d327fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:56.585 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.602+0000 7f1d327fc640 1 -- 172.21.15.112:0/1697501803 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1d34001830 con 0x7f1d3c1457f0 2024-03-29T10:58:56.585 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.602+0000 7f1d327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:56.585 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.602+0000 7f1d327fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:56.585 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.602+0000 7f1d327fc640 10 monclient(hunting): dump: 2024-03-29T10:58:56.585 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:56.585 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:56.585 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:56.585 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:56.585 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:56.585 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:56.586 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:56.586 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:56.586 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:56.586 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:56.586 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.602+0000 7f1d327fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:56.586 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.602+0000 7f1d327fc640 1 -- 172.21.15.112:0/1697501803 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1670989148 0 0) 0x7f1d3c071110 con 0x7f1d3c13f650 2024-03-29T10:58:56.586 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.602+0000 7f1d327fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:56.586 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.602+0000 7f1d327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:56.586 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.602+0000 7f1d327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:56.586 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.602+0000 7f1d327fc640 10 monclient(hunting): my global_id is 4500 2024-03-29T10:58:56.586 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.602+0000 7f1d327fc640 1 -- 172.21.15.112:0/1697501803 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d20004eb0 con 0x7f1d3c13f650 2024-03-29T10:58:56.586 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.602+0000 7f1d327fc640 1 -- 172.21.15.112:0/1697501803 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1423279742 0 0) 0x7f1d3c138100 con 0x7f1d3c1457f0 2024-03-29T10:58:56.586 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.602+0000 7f1d327fc640 1 -- 172.21.15.112:0/1697501803 >> v1:172.21.15.112:6790/0 conn(0x7f1d3c0a9d70 legacy=0x7f1d3c138dc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:58:56.587 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d327fc640 1 -- 172.21.15.112:0/1697501803 >> v1:172.21.15.112:6789/0 conn(0x7f1d3c13f650 legacy=0x7f1d3c0715a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:56.587 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d327fc640 10 monclient: _finish_hunting 0 2024-03-29T10:58:56.587 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d327fc640 1 monclient: found mon.b 2024-03-29T10:58:56.587 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:56.587 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d327fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:58:56.587 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d327fc640 1 -- 172.21.15.112:0/1697501803 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3c1386c0 con 0x7f1d3c1457f0 2024-03-29T10:58:56.587 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d327fc640 10 monclient: _finish_auth 0 2024-03-29T10:58:56.587 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d327fc640 10 monclient: _check_auth_tickets 2024-03-29T10:58:56.587 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:56.587 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d327fc640 10 client.?.objecter ms_handle_connect 0x7f1d3c0a9d70 2024-03-29T10:58:56.588 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d327fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:56.588 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d43fc6640 5 monclient: authenticate success, global_id 4483 2024-03-29T10:58:56.588 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d43fc6640 10 monclient: _renew_subs 2024-03-29T10:58:56.588 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d43fc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:58:56.588 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d43fc6640 1 -- 172.21.15.112:0/1697501803 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d3c1389a0 con 0x7f1d3c1457f0 2024-03-29T10:58:56.588 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d43fc6640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:56.588 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d43fc6640 10 monclient: _renew_subs 2024-03-29T10:58:56.588 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d43fc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:58:56.588 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d43fc6640 1 -- 172.21.15.112:0/1697501803 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d3c070890 con 0x7f1d3c1457f0 2024-03-29T10:58:56.588 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d43fc6640 1 librados: init done 2024-03-29T10:58:56.588 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d327fc640 1 -- 172.21.15.112:0/1697501803 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d34004ff0 con 0x7f1d3c1457f0 2024-03-29T10:58:56.589 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:56.589 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d327fc640 1 -- 172.21.15.112:0/1697501803 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1d34001990 con 0x7f1d3c1457f0 2024-03-29T10:58:56.589 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:56.590 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:56.590 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d327fc640 10 monclient: dump: 2024-03-29T10:58:56.590 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:58:56.590 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:58:56.590 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:56.590 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:58:56.590 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:56.590 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:58:56.590 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:58:56.590 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:58:56.590 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:58:56.590 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:58:56.590 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.606+0000 7f1d337fe640 4 set_mon_vals no callback set 2024-03-29T10:58:56.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.607+0000 7f1d43fc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:56.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.607+0000 7f1d43fc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:56.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.607+0000 7f1d43fc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:58:56.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.607+0000 7f1d43fc6640 1 -- 172.21.15.112:0/1697501803 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d24005080 con 0x7f1d3c1457f0 2024-03-29T10:58:56.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.609+0000 7f1d327fc640 1 -- 172.21.15.112:0/1697501803 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f1d3401b170 con 0x7f1d3c1457f0 2024-03-29T10:58:56.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.609+0000 7f1d327fc640 1 -- 172.21.15.112:0/1697501803 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f1d340850e0 con 0x7f1d3c1457f0 2024-03-29T10:58:56.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.609+0000 7f1d327fc640 10 client.4483.objecter ms_dispatch 0x7f1d3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:56.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.609+0000 7f1d327fc640 3 client.4483.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:56.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.609+0000 7f1d327fc640 3 client.4483.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:56.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.609+0000 7f1d327fc640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-29T10:58:56.592 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.609+0000 7f1d41d62640 10 monclient: get_auth_request con 0x7f1d2006dc00 auth_method 0 2024-03-29T10:58:56.592 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.610+0000 7f1d327fc640 10 client.4483.objecter ms_handle_connect 0x7f1d2006dc00 2024-03-29T10:58:56.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.618+0000 7f1d327fc640 1 -- 172.21.15.112:0/1697501803 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1d340507f0 con 0x7f1d3c1457f0 2024-03-29T10:58:56.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.618+0000 7f1d327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:56.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.618+0000 7f1d327fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:56.922 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.941+0000 7f1d43fc6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:56.922 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.941+0000 7f1d43fc6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:56.922 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.941+0000 7f1d43fc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:58:56.922 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.941+0000 7f1d43fc6640 1 -- 172.21.15.112:0/1697501803 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d24005330 con 0x7f1d3c1457f0 2024-03-29T10:58:56.925 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:58:56.925 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:56.925 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.944+0000 7f1d327fc640 1 -- 172.21.15.112:0/1697501803 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f1d340528f0 con 0x7f1d3c1457f0 2024-03-29T10:58:56.925 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.944+0000 7f1d327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:56.925 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.944+0000 7f1d327fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:56.929 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.948+0000 7f1d43fc6640 10 librados: watch_flush enter 2024-03-29T10:58:56.929 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.948+0000 7f1d43fc6640 10 librados: watch_flush exit 2024-03-29T10:58:56.930 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.948+0000 7f1d43fc6640 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-29T10:58:56.930 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.948+0000 7f1d43fc6640 10 client.4483.objecter successfully canceled tick 2024-03-29T10:58:56.930 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.948+0000 7f1d43fc6640 1 -- 172.21.15.112:0/1697501803 >> v1:172.21.15.118:6816/3181073408 conn(0x7f1d2006dc00 legacy=0x7f1d200700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:56.930 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.948+0000 7f1d43fc6640 10 monclient: shutdown 2024-03-29T10:58:56.930 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.948+0000 7f1d43fc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:56.930 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.948+0000 7f1d43fc6640 1 -- 172.21.15.112:0/1697501803 >> v1:172.21.15.118:6789/0 conn(0x7f1d3c1457f0 legacy=0x7f1d3c1e6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:56.930 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.948+0000 7f1d43fc6640 1 -- 172.21.15.112:0/1697501803 shutdown_connections 2024-03-29T10:58:56.930 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.948+0000 7f1d43fc6640 1 -- 172.21.15.112:0/1697501803 >> 172.21.15.112:0/1697501803 conn(0x7f1d3c066a00 msgr2=0x7f1d3c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:56.930 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.949+0000 7f1d43fc6640 1 -- 172.21.15.112:0/1697501803 shutdown_connections 2024-03-29T10:58:56.930 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.949+0000 7f1d43fc6640 1 -- 172.21.15.112:0/1697501803 wait complete. 2024-03-29T10:58:56.930 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:58:56.949+0000 7f1d43fc6640 1 librados: shutdown 2024-03-29T10:58:56.945 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:03.751 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:03.905 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.924+0000 7fb34080c640 10 monclient: get_monmap_and_config 2024-03-29T10:59:03.905 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.924+0000 7fb34080c640 10 monclient: build_initial_monmap 2024-03-29T10:59:03.905 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.924+0000 7fb34080c640 10 monclient: monmap: 2024-03-29T10:59:03.905 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:59:03.905 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:03.905 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:59:03.925603+0000 2024-03-29T10:59:03.905 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:59:03.925603+0000 2024-03-29T10:59:03.906 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:03.906 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:03.906 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:59:03.906 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:59:03.906 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:59:03.906 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:03.907 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.927+0000 7fb34080c640 1 Processor -- start 2024-03-29T10:59:03.908 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.927+0000 7fb34080c640 1 -- start start 2024-03-29T10:59:03.908 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.927+0000 7fb34080c640 10 monclient: init 2024-03-29T10:59:03.908 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.928+0000 7fb34080c640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:03.909 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.928+0000 7fb34080c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:59:03.909 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.928+0000 7fb34080c640 10 monclient(hunting): picked mon.noname-a con 0x7fb3340389e0 addr v1:172.21.15.112:6789/0 2024-03-29T10:59:03.909 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.928+0000 7fb34080c640 10 monclient(hunting): picked mon.noname-c con 0x7fb334038dc0 addr v1:172.21.15.118:6789/0 2024-03-29T10:59:03.909 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.928+0000 7fb34080c640 10 monclient(hunting): picked mon.noname-b con 0x7fb334030660 addr v1:172.21.15.112:6790/0 2024-03-29T10:59:03.909 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.928+0000 7fb34080c640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3340c54c0 con 0x7fb3340389e0 2024-03-29T10:59:03.909 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.928+0000 7fb34080c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb334036430 con 0x7fb334038dc0 2024-03-29T10:59:03.909 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.928+0000 7fb34080c640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb334039840 con 0x7fb334030660 2024-03-29T10:59:03.909 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.928+0000 7fb34080c640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:03.909 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.928+0000 7fb34080c640 10 monclient(hunting): authenticate will time out at 1299.233035s 2024-03-29T10:59:03.912 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.932+0000 7fb33ad9d640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fb3340389e0 0x7fb334033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59587/0 (socket says 172.21.15.112:59587) 2024-03-29T10:59:03.912 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.932+0000 7fb33ad9d640 1 -- 172.21.15.112:0/1569627127 learned_addr learned my addr 172.21.15.112:0/1569627127 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:59:03.913 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.932+0000 7fb339d9b640 1 -- 172.21.15.112:0/1569627127 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320904223 0 0) 0x7fb334039840 con 0x7fb334030660 2024-03-29T10:59:03.913 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.932+0000 7fb339d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:59:03.913 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.932+0000 7fb339d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:03.913 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.932+0000 7fb339d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:03.913 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.932+0000 7fb339d9b640 10 monclient(hunting): my global_id is 4529 2024-03-29T10:59:03.913 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.932+0000 7fb339d9b640 1 -- 172.21.15.112:0/1569627127 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb324003520 con 0x7fb334030660 2024-03-29T10:59:03.913 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.933+0000 7fb339d9b640 1 -- 172.21.15.112:0/1569627127 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949557348 0 0) 0x7fb3340c54c0 con 0x7fb3340389e0 2024-03-29T10:59:03.913 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.933+0000 7fb339d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:59:03.913 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.933+0000 7fb339d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:03.913 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.933+0000 7fb339d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:03.913 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.933+0000 7fb339d9b640 10 monclient(hunting): my global_id is 4503 2024-03-29T10:59:03.913 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.933+0000 7fb339d9b640 1 -- 172.21.15.112:0/1569627127 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb334039840 con 0x7fb3340389e0 2024-03-29T10:59:03.917 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.936+0000 7fb339d9b640 1 -- 172.21.15.112:0/1569627127 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3278117509 0 0) 0x7fb324003520 con 0x7fb334030660 2024-03-29T10:59:03.918 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.936+0000 7fb339d9b640 1 -- 172.21.15.112:0/1569627127 >> v1:172.21.15.118:6789/0 conn(0x7fb334038dc0 legacy=0x7fb3340cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:59:03.918 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.936+0000 7fb339d9b640 1 -- 172.21.15.112:0/1569627127 >> v1:172.21.15.112:6789/0 conn(0x7fb3340389e0 legacy=0x7fb334033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:03.918 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.936+0000 7fb339d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:59:03.918 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.936+0000 7fb339d9b640 1 monclient: found mon.noname-b 2024-03-29T10:59:03.918 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.936+0000 7fb339d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:03.918 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.936+0000 7fb339d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:59:03.918 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.936+0000 7fb339d9b640 1 -- 172.21.15.112:0/1569627127 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb334035690 con 0x7fb334030660 2024-03-29T10:59:03.918 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.936+0000 7fb339d9b640 10 monclient: _finish_auth 0 2024-03-29T10:59:03.918 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.936+0000 7fb339d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:59:03.918 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.936+0000 7fb339d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:03.918 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.936+0000 7fb339d9b640 1 -- 172.21.15.112:0/1569627127 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb32c001690 con 0x7fb334030660 2024-03-29T10:59:03.918 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.936+0000 7fb339d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:03.918 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.936+0000 7fb339d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:03.919 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.936+0000 7fb339d9b640 10 monclient: dump: 2024-03-29T10:59:03.919 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:03.919 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:03.919 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:03.919 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:03.919 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:03.919 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:03.919 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:03.919 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:03.919 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:03.919 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:03.919 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.936+0000 7fb34080c640 5 monclient: authenticate success, global_id 4529 2024-03-29T10:59:03.919 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.936+0000 7fb34080c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:03.921 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.938+0000 7fb339d9b640 1 -- 172.21.15.112:0/1569627127 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb32c0017f0 con 0x7fb334030660 2024-03-29T10:59:03.921 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.938+0000 7fb339d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:03.921 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.938+0000 7fb339d9b640 1 -- 172.21.15.112:0/1569627127 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb32c001c40 con 0x7fb334030660 2024-03-29T10:59:03.921 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.938+0000 7fb34080c640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:03.921 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.938+0000 7fb34080c640 4 set_mon_vals no callback set 2024-03-29T10:59:03.921 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.939+0000 7fb339d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:03.921 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.939+0000 7fb339d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:03.921 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.939+0000 7fb339d9b640 10 monclient: dump: 2024-03-29T10:59:03.921 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:03.921 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:03.921 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:03.921 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:03.921 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:03.921 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:03.922 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:03.922 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:03.922 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:03.922 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:03.922 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.939+0000 7fb34080c640 10 monclient: shutdown 2024-03-29T10:59:03.922 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.939+0000 7fb34080c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:03.922 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.939+0000 7fb34080c640 1 -- 172.21.15.112:0/1569627127 >> v1:172.21.15.112:6790/0 conn(0x7fb334030660 legacy=0x7fb3340d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:03.922 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.939+0000 7fb34080c640 1 -- 172.21.15.112:0/1569627127 shutdown_connections 2024-03-29T10:59:03.922 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.939+0000 7fb34080c640 1 -- 172.21.15.112:0/1569627127 >> 172.21.15.112:0/1569627127 conn(0x7fb334005c00 msgr2=0x7fb334006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:03.922 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.940+0000 7fb34080c640 1 -- 172.21.15.112:0/1569627127 shutdown_connections 2024-03-29T10:59:03.922 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.940+0000 7fb34080c640 1 -- 172.21.15.112:0/1569627127 wait complete. 2024-03-29T10:59:03.925 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.941+0000 7fb34080c640 10 monclient: build_initial_monmap 2024-03-29T10:59:03.925 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.941+0000 7fb34080c640 10 monclient: monmap: 2024-03-29T10:59:03.925 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:59:03.925 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:03.925 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:59:03.926 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:59:03.926 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:03.926 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:03.926 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:59:03.926 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:59:03.926 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:59:03.926 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:03.926 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.941+0000 7fb34080c640 1 librados: starting msgr at 2024-03-29T10:59:03.926 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.941+0000 7fb34080c640 1 librados: starting objecter 2024-03-29T10:59:03.926 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.941+0000 7fb34080c640 1 Processor -- start 2024-03-29T10:59:03.926 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.942+0000 7fb34080c640 1 -- start start 2024-03-29T10:59:03.926 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.942+0000 7fb34080c640 1 librados: setting wanted keys 2024-03-29T10:59:03.926 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.942+0000 7fb34080c640 1 librados: calling monclient init 2024-03-29T10:59:03.927 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.942+0000 7fb34080c640 10 monclient: init 2024-03-29T10:59:03.927 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.942+0000 7fb34080c640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:03.927 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.942+0000 7fb34080c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:59:03.927 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.942+0000 7fb34080c640 10 monclient(hunting): picked mon.noname-c con 0x7fb334030660 addr v1:172.21.15.112:6790/0 2024-03-29T10:59:03.927 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.942+0000 7fb34080c640 10 monclient(hunting): picked mon.noname-b con 0x7fb3340389e0 addr v1:172.21.15.118:6789/0 2024-03-29T10:59:03.927 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.942+0000 7fb34080c640 10 monclient(hunting): picked mon.noname-a con 0x7fb3340c7920 addr v1:172.21.15.112:6789/0 2024-03-29T10:59:03.927 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.942+0000 7fb34080c640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33400a470 con 0x7fb3340c7920 2024-03-29T10:59:03.927 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.942+0000 7fb34080c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33416d190 con 0x7fb3340389e0 2024-03-29T10:59:03.927 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.942+0000 7fb34080c640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33416ce20 con 0x7fb334030660 2024-03-29T10:59:03.927 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.942+0000 7fb34080c640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:03.927 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.942+0000 7fb34080c640 10 monclient(hunting): authenticate will time out at 1299.247060s 2024-03-29T10:59:03.934 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.949+0000 7fb33ad9d640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fb334030660 0x7fb33416c710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:34941/0 (socket says 172.21.15.112:34941) 2024-03-29T10:59:03.934 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.949+0000 7fb33ad9d640 1 -- 172.21.15.112:0/1575527335 learned_addr learned my addr 172.21.15.112:0/1575527335 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:59:03.934 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.949+0000 7fb3237fe640 10 client.?.objecter ms_handle_connect 0x7fb334030660 2024-03-29T10:59:03.934 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.949+0000 7fb3237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:03.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.949+0000 7fb3237fe640 1 -- 172.21.15.112:0/1575527335 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2753959241 0 0) 0x7fb33416ce20 con 0x7fb334030660 2024-03-29T10:59:03.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.949+0000 7fb3237fe640 10 monclient(hunting): _negotiate 2024-03-29T10:59:03.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.949+0000 7fb3237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:03.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.949+0000 7fb3237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:03.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.949+0000 7fb3237fe640 10 monclient(hunting): my global_id is 4532 2024-03-29T10:59:03.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.949+0000 7fb3237fe640 1 -- 172.21.15.112:0/1575527335 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb328004eb0 con 0x7fb334030660 2024-03-29T10:59:03.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.950+0000 7fb3237fe640 10 client.?.objecter ms_handle_connect 0x7fb3340c7920 2024-03-29T10:59:03.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.950+0000 7fb3237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:03.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.950+0000 7fb3237fe640 10 client.?.objecter ms_handle_connect 0x7fb3340389e0 2024-03-29T10:59:03.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.950+0000 7fb3237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:03.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.950+0000 7fb3237fe640 1 -- 172.21.15.112:0/1575527335 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3655274045 0 0) 0x7fb33400a470 con 0x7fb3340c7920 2024-03-29T10:59:03.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.950+0000 7fb3237fe640 10 monclient(hunting): _negotiate 2024-03-29T10:59:03.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.950+0000 7fb3237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:03.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.950+0000 7fb3237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:03.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.950+0000 7fb3237fe640 10 monclient(hunting): my global_id is 4506 2024-03-29T10:59:03.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.950+0000 7fb3237fe640 1 -- 172.21.15.112:0/1575527335 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb33416ce20 con 0x7fb3340c7920 2024-03-29T10:59:03.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.950+0000 7fb3237fe640 1 -- 172.21.15.112:0/1575527335 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390909764 0 0) 0x7fb33416d190 con 0x7fb3340389e0 2024-03-29T10:59:03.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.950+0000 7fb3237fe640 10 monclient(hunting): _negotiate 2024-03-29T10:59:03.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.950+0000 7fb3237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:03.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.950+0000 7fb3237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:03.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.950+0000 7fb3237fe640 10 monclient(hunting): my global_id is 4486 2024-03-29T10:59:03.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.950+0000 7fb3237fe640 1 -- 172.21.15.112:0/1575527335 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb33400a470 con 0x7fb3340389e0 2024-03-29T10:59:03.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb3237fe640 1 -- 172.21.15.112:0/1575527335 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1840238812 0 0) 0x7fb33416ce20 con 0x7fb3340c7920 2024-03-29T10:59:03.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb3237fe640 1 -- 172.21.15.112:0/1575527335 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb33416d190 con 0x7fb3340c7920 2024-03-29T10:59:03.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb3237fe640 1 -- 172.21.15.112:0/1575527335 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb32c001b00 con 0x7fb3340c7920 2024-03-29T10:59:03.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb3237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:03.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb3237fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:59:03.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb3237fe640 10 monclient(hunting): dump: 2024-03-29T10:59:03.936 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:03.937 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:03.937 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:03.937 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:03.937 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:03.937 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:03.937 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:03.937 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:03.937 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:03.937 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:03.937 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb3237fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:03.937 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb3237fe640 1 -- 172.21.15.112:0/1575527335 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2153519620 0 0) 0x7fb328004eb0 con 0x7fb334030660 2024-03-29T10:59:03.937 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb3237fe640 1 -- 172.21.15.112:0/1575527335 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb33416ce20 con 0x7fb334030660 2024-03-29T10:59:03.937 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb3237fe640 1 -- 172.21.15.112:0/1575527335 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb330001a50 con 0x7fb334030660 2024-03-29T10:59:03.937 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb3237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:03.937 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb3237fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:03.937 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb3237fe640 10 monclient(hunting): dump: 2024-03-29T10:59:03.938 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:03.938 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:03.938 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:03.938 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:03.938 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:03.938 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:03.938 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:03.938 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:03.938 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:03.938 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:03.938 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb3237fe640 1 -- 172.21.15.112:0/1575527335 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 316856609 0 0) 0x7fb33416ce20 con 0x7fb334030660 2024-03-29T10:59:03.938 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb3237fe640 1 -- 172.21.15.112:0/1575527335 >> v1:172.21.15.118:6789/0 conn(0x7fb3340389e0 legacy=0x7fb3341662d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:03.938 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb3237fe640 1 -- 172.21.15.112:0/1575527335 >> v1:172.21.15.112:6789/0 conn(0x7fb3340c7920 legacy=0x7fb334169210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:03.938 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb3237fe640 10 monclient: _finish_hunting 0 2024-03-29T10:59:03.938 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb3237fe640 1 monclient: found mon.c 2024-03-29T10:59:03.939 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb3237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:03.939 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb3237fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:59:03.939 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb3237fe640 1 -- 172.21.15.112:0/1575527335 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3341669e0 con 0x7fb334030660 2024-03-29T10:59:03.939 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb3237fe640 10 monclient: _finish_auth 0 2024-03-29T10:59:03.939 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb3237fe640 10 monclient: _check_auth_tickets 2024-03-29T10:59:03.939 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb3237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:03.939 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb34080c640 5 monclient: authenticate success, global_id 4532 2024-03-29T10:59:03.939 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb34080c640 10 monclient: _renew_subs 2024-03-29T10:59:03.939 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb34080c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:59:03.939 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb34080c640 1 -- 172.21.15.112:0/1575527335 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb334166c90 con 0x7fb334030660 2024-03-29T10:59:03.939 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb34080c640 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:03.939 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb34080c640 10 monclient: _renew_subs 2024-03-29T10:59:03.939 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.951+0000 7fb34080c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:59:03.939 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.952+0000 7fb34080c640 1 -- 172.21.15.112:0/1575527335 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb334169b00 con 0x7fb334030660 2024-03-29T10:59:03.939 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.952+0000 7fb34080c640 1 librados: init done 2024-03-29T10:59:03.940 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.953+0000 7fb3237fe640 1 -- 172.21.15.112:0/1575527335 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3300047e0 con 0x7fb334030660 2024-03-29T10:59:03.940 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.953+0000 7fb3237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:03.940 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.953+0000 7fb3237fe640 1 -- 172.21.15.112:0/1575527335 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb330004af0 con 0x7fb334030660 2024-03-29T10:59:03.940 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.953+0000 7fb3237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:03.940 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.953+0000 7fb3237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:03.940 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.953+0000 7fb3237fe640 10 monclient: dump: 2024-03-29T10:59:03.940 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:03.940 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:03.940 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:03.940 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:03.940 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:03.940 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:03.940 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:03.940 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:03.941 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:03.941 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:03.941 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.954+0000 7fb33959a640 4 set_mon_vals no callback set 2024-03-29T10:59:03.942 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.955+0000 7fb3237fe640 1 -- 172.21.15.112:0/1575527335 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fb33001aac0 con 0x7fb334030660 2024-03-29T10:59:03.942 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.955+0000 7fb34080c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:03.942 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.956+0000 7fb34080c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:03.942 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.956+0000 7fb34080c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:59:03.942 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.956+0000 7fb34080c640 1 -- 172.21.15.112:0/1575527335 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31c005080 con 0x7fb334030660 2024-03-29T10:59:03.942 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.956+0000 7fb3237fe640 1 -- 172.21.15.112:0/1575527335 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7fb330084b70 con 0x7fb334030660 2024-03-29T10:59:03.943 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.956+0000 7fb3237fe640 10 client.4532.objecter ms_dispatch 0x7fb33c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:03.943 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.956+0000 7fb3237fe640 3 client.4532.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:03.943 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.956+0000 7fb3237fe640 3 client.4532.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:03.943 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.956+0000 7fb3237fe640 20 client.4532.objecter dump_active .. 0 homeless 2024-03-29T10:59:03.943 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.957+0000 7fb33a59c640 10 monclient: get_auth_request con 0x7fb32806de70 auth_method 0 2024-03-29T10:59:03.943 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.957+0000 7fb3237fe640 10 client.4532.objecter ms_handle_connect 0x7fb32806de70 2024-03-29T10:59:03.948 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.968+0000 7fb3237fe640 1 -- 172.21.15.112:0/1575527335 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb330050280 con 0x7fb334030660 2024-03-29T10:59:03.948 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.968+0000 7fb3237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:03.948 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:03.968+0000 7fb3237fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:04.268 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:04.287+0000 7fb34080c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:04.268 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:04.287+0000 7fb34080c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:04.268 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:04.287+0000 7fb34080c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:59:04.268 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:04.287+0000 7fb34080c640 1 -- 172.21.15.112:0/1575527335 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb31c005330 con 0x7fb334030660 2024-03-29T10:59:04.269 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:04.288+0000 7fb3237fe640 1 -- 172.21.15.112:0/1575527335 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7fb330052380 con 0x7fb334030660 2024-03-29T10:59:04.269 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:04.288+0000 7fb3237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:04.269 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:04.288+0000 7fb3237fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:04.269 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:59:04.269 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:04.272 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:04.292+0000 7fb34080c640 10 librados: watch_flush enter 2024-03-29T10:59:04.274 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:04.294+0000 7fb34080c640 10 librados: watch_flush exit 2024-03-29T10:59:04.274 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:04.294+0000 7fb34080c640 20 client.4532.objecter shutdown clearing up homeless session... 2024-03-29T10:59:04.275 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:04.294+0000 7fb34080c640 10 client.4532.objecter successfully canceled tick 2024-03-29T10:59:04.275 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:04.294+0000 7fb34080c640 1 -- 172.21.15.112:0/1575527335 >> v1:172.21.15.118:6816/3181073408 conn(0x7fb32806de70 legacy=0x7fb328070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:04.275 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:04.294+0000 7fb34080c640 10 monclient: shutdown 2024-03-29T10:59:04.275 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:04.294+0000 7fb34080c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:04.275 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:04.294+0000 7fb34080c640 1 -- 172.21.15.112:0/1575527335 >> v1:172.21.15.112:6790/0 conn(0x7fb334030660 legacy=0x7fb33416c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:04.275 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:04.294+0000 7fb34080c640 1 -- 172.21.15.112:0/1575527335 shutdown_connections 2024-03-29T10:59:04.275 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:04.294+0000 7fb34080c640 1 -- 172.21.15.112:0/1575527335 >> 172.21.15.112:0/1575527335 conn(0x7fb334005c00 msgr2=0x7fb334034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:04.275 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:04.294+0000 7fb34080c640 1 -- 172.21.15.112:0/1575527335 shutdown_connections 2024-03-29T10:59:04.275 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:04.294+0000 7fb34080c640 1 -- 172.21.15.112:0/1575527335 wait complete. 2024-03-29T10:59:04.275 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:04.294+0000 7fb34080c640 1 librados: shutdown 2024-03-29T10:59:04.295 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:11.101 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:11.250 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.269+0000 7f0b68ac8640 10 monclient: get_monmap_and_config 2024-03-29T10:59:11.250 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.269+0000 7f0b68ac8640 10 monclient: build_initial_monmap 2024-03-29T10:59:11.250 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.269+0000 7f0b68ac8640 10 monclient: monmap: 2024-03-29T10:59:11.250 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:59:11.251 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:11.251 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:59:11.270342+0000 2024-03-29T10:59:11.251 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:59:11.270342+0000 2024-03-29T10:59:11.251 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:11.251 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:11.251 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:59:11.251 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:59:11.251 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:59:11.251 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:11.251 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.270+0000 7f0b68ac8640 1 Processor -- start 2024-03-29T10:59:11.252 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.270+0000 7f0b68ac8640 1 -- start start 2024-03-29T10:59:11.252 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.270+0000 7f0b68ac8640 10 monclient: init 2024-03-29T10:59:11.253 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.271+0000 7f0b68ac8640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:11.253 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.271+0000 7f0b68ac8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:59:11.253 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.271+0000 7f0b68ac8640 10 monclient(hunting): picked mon.noname-b con 0x7f0b640a9ac0 addr v1:172.21.15.112:6790/0 2024-03-29T10:59:11.253 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.271+0000 7f0b68ac8640 10 monclient(hunting): picked mon.noname-a con 0x7f0b6413ee90 addr v1:172.21.15.112:6789/0 2024-03-29T10:59:11.253 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.271+0000 7f0b68ac8640 10 monclient(hunting): picked mon.noname-c con 0x7f0b640b4100 addr v1:172.21.15.118:6789/0 2024-03-29T10:59:11.253 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.271+0000 7f0b68ac8640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b640af6f0 con 0x7f0b6413ee90 2024-03-29T10:59:11.253 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.271+0000 7f0b68ac8640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b640aac80 con 0x7f0b640b4100 2024-03-29T10:59:11.253 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.272+0000 7f0b68ac8640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b640aaf60 con 0x7f0b640a9ac0 2024-03-29T10:59:11.253 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.273+0000 7f0b68ac8640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:11.253 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.273+0000 7f0b68ac8640 10 monclient(hunting): authenticate will time out at 1306.577192s 2024-03-29T10:59:11.253 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.273+0000 7f0b6259c640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f0b640a9ac0 0x7f0b640adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:33516/0 (socket says 172.21.15.112:33516) 2024-03-29T10:59:11.253 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.273+0000 7f0b6259c640 1 -- 172.21.15.112:0/2876798787 learned_addr learned my addr 172.21.15.112:0/2876798787 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:59:11.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.274+0000 7f0b6159a640 1 -- 172.21.15.112:0/2876798787 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898166038 0 0) 0x7f0b640aaf60 con 0x7f0b640a9ac0 2024-03-29T10:59:11.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.274+0000 7f0b6159a640 10 monclient(hunting): _negotiate 2024-03-29T10:59:11.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.274+0000 7f0b6159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:11.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.274+0000 7f0b6159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:11.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.274+0000 7f0b6159a640 10 monclient(hunting): my global_id is 4535 2024-03-29T10:59:11.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.274+0000 7f0b6159a640 1 -- 172.21.15.112:0/2876798787 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b4c003520 con 0x7f0b640a9ac0 2024-03-29T10:59:11.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.275+0000 7f0b6159a640 1 -- 172.21.15.112:0/2876798787 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3180838057 0 0) 0x7f0b4c003520 con 0x7f0b640a9ac0 2024-03-29T10:59:11.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.275+0000 7f0b6159a640 1 -- 172.21.15.112:0/2876798787 >> v1:172.21.15.118:6789/0 conn(0x7f0b640b4100 legacy=0x7f0b6414a390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:59:11.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.275+0000 7f0b6159a640 1 -- 172.21.15.112:0/2876798787 >> v1:172.21.15.112:6789/0 conn(0x7f0b6413ee90 legacy=0x7f0b6413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:59:11.256 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.275+0000 7f0b6159a640 10 monclient: _finish_hunting 0 2024-03-29T10:59:11.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.275+0000 7f0b6159a640 1 monclient: found mon.noname-b 2024-03-29T10:59:11.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.275+0000 7f0b6159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:11.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.275+0000 7f0b6159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:59:11.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.275+0000 7f0b6159a640 1 -- 172.21.15.112:0/2876798787 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6413e970 con 0x7f0b640a9ac0 2024-03-29T10:59:11.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.275+0000 7f0b6159a640 10 monclient: _finish_auth 0 2024-03-29T10:59:11.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.275+0000 7f0b6159a640 10 monclient: _check_auth_tickets 2024-03-29T10:59:11.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.275+0000 7f0b6159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:11.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.275+0000 7f0b6159a640 1 -- 172.21.15.112:0/2876798787 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f0b5c001a10 con 0x7f0b640a9ac0 2024-03-29T10:59:11.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.275+0000 7f0b6159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:11.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.275+0000 7f0b6159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:11.257 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.275+0000 7f0b6159a640 10 monclient: dump: 2024-03-29T10:59:11.257 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:11.257 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:11.257 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:11.257 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:11.258 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:11.258 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:11.258 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:11.258 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:11.258 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:11.258 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:11.258 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.275+0000 7f0b68ac8640 5 monclient: authenticate success, global_id 4535 2024-03-29T10:59:11.258 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.275+0000 7f0b68ac8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:11.259 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.277+0000 7f0b6159a640 1 -- 172.21.15.112:0/2876798787 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b5c001c70 con 0x7f0b640a9ac0 2024-03-29T10:59:11.259 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.277+0000 7f0b6159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:11.260 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.277+0000 7f0b6159a640 1 -- 172.21.15.112:0/2876798787 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f0b5c0020c0 con 0x7f0b640a9ac0 2024-03-29T10:59:11.260 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.277+0000 7f0b6159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:11.260 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.277+0000 7f0b6159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:11.260 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.277+0000 7f0b6159a640 10 monclient: dump: 2024-03-29T10:59:11.260 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:11.260 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:11.260 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:11.260 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:11.260 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:11.260 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:11.260 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:11.260 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:11.260 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:11.260 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:11.260 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.277+0000 7f0b68ac8640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:11.260 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.277+0000 7f0b68ac8640 4 set_mon_vals no callback set 2024-03-29T10:59:11.261 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.278+0000 7f0b68ac8640 10 monclient: shutdown 2024-03-29T10:59:11.261 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.278+0000 7f0b68ac8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:11.261 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.278+0000 7f0b68ac8640 1 -- 172.21.15.112:0/2876798787 >> v1:172.21.15.112:6790/0 conn(0x7f0b640a9ac0 legacy=0x7f0b640adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:11.261 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.278+0000 7f0b68ac8640 1 -- 172.21.15.112:0/2876798787 shutdown_connections 2024-03-29T10:59:11.261 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.278+0000 7f0b68ac8640 1 -- 172.21.15.112:0/2876798787 >> 172.21.15.112:0/2876798787 conn(0x7f0b640b35e0 msgr2=0x7f0b640b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:11.261 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.278+0000 7f0b68ac8640 1 -- 172.21.15.112:0/2876798787 shutdown_connections 2024-03-29T10:59:11.261 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.278+0000 7f0b68ac8640 1 -- 172.21.15.112:0/2876798787 wait complete. 2024-03-29T10:59:11.262 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.279+0000 7f0b68ac8640 10 monclient: build_initial_monmap 2024-03-29T10:59:11.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.279+0000 7f0b68ac8640 10 monclient: monmap: 2024-03-29T10:59:11.263 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:59:11.263 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:11.263 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:59:11.263 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:59:11.263 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:11.263 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:11.263 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:59:11.263 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:59:11.263 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:59:11.263 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:11.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.280+0000 7f0b68ac8640 1 librados: starting msgr at 2024-03-29T10:59:11.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.280+0000 7f0b68ac8640 1 librados: starting objecter 2024-03-29T10:59:11.263 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.280+0000 7f0b68ac8640 1 Processor -- start 2024-03-29T10:59:11.264 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.280+0000 7f0b68ac8640 1 -- start start 2024-03-29T10:59:11.264 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.280+0000 7f0b68ac8640 1 librados: setting wanted keys 2024-03-29T10:59:11.264 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.280+0000 7f0b68ac8640 1 librados: calling monclient init 2024-03-29T10:59:11.264 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.280+0000 7f0b68ac8640 10 monclient: init 2024-03-29T10:59:11.264 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.281+0000 7f0b68ac8640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:11.264 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.281+0000 7f0b68ac8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:59:11.264 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.281+0000 7f0b68ac8640 10 monclient(hunting): picked mon.noname-a con 0x7f0b640a9ac0 addr v1:172.21.15.112:6789/0 2024-03-29T10:59:11.264 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.281+0000 7f0b68ac8640 10 monclient(hunting): picked mon.noname-c con 0x7f0b640b4100 addr v1:172.21.15.112:6790/0 2024-03-29T10:59:11.264 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.281+0000 7f0b68ac8640 10 monclient(hunting): picked mon.noname-b con 0x7f0b6413ee90 addr v1:172.21.15.118:6789/0 2024-03-29T10:59:11.264 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.281+0000 7f0b68ac8640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b64150b70 con 0x7f0b640a9ac0 2024-03-29T10:59:11.264 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.281+0000 7f0b68ac8640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b64141290 con 0x7f0b6413ee90 2024-03-29T10:59:11.264 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.281+0000 7f0b68ac8640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b64141570 con 0x7f0b640b4100 2024-03-29T10:59:11.264 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.281+0000 7f0b68ac8640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:11.265 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.281+0000 7f0b68ac8640 10 monclient(hunting): authenticate will time out at 1306.586073s 2024-03-29T10:59:11.265 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.282+0000 7f0b61d9b640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f0b640b4100 0x7f0b6414fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35484/0 (socket says 172.21.15.112:35484) 2024-03-29T10:59:11.265 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.282+0000 7f0b61d9b640 1 -- 172.21.15.112:0/3930335576 learned_addr learned my addr 172.21.15.112:0/3930335576 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:59:11.266 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.285+0000 7f0b52ffd640 10 client.?.objecter ms_handle_connect 0x7f0b640a9ac0 2024-03-29T10:59:11.266 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.285+0000 7f0b52ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:11.267 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.285+0000 7f0b52ffd640 1 -- 172.21.15.112:0/3930335576 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894714121 0 0) 0x7f0b64150b70 con 0x7f0b640a9ac0 2024-03-29T10:59:11.267 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.285+0000 7f0b52ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:59:11.267 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.285+0000 7f0b52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:11.267 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.285+0000 7f0b52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:11.267 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.285+0000 7f0b52ffd640 10 monclient(hunting): my global_id is 4509 2024-03-29T10:59:11.267 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.285+0000 7f0b52ffd640 1 -- 172.21.15.112:0/3930335576 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b44004eb0 con 0x7f0b640a9ac0 2024-03-29T10:59:11.267 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.286+0000 7f0b52ffd640 1 -- 172.21.15.112:0/3930335576 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1390344452 0 0) 0x7f0b44004eb0 con 0x7f0b640a9ac0 2024-03-29T10:59:11.267 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.286+0000 7f0b52ffd640 1 -- 172.21.15.112:0/3930335576 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b64150b70 con 0x7f0b640a9ac0 2024-03-29T10:59:11.267 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.286+0000 7f0b52ffd640 1 -- 172.21.15.112:0/3930335576 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f0b5c001830 con 0x7f0b640a9ac0 2024-03-29T10:59:11.267 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.286+0000 7f0b52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:11.267 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.286+0000 7f0b52ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:59:11.267 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.286+0000 7f0b52ffd640 10 monclient(hunting): dump: 2024-03-29T10:59:11.267 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:11.267 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:11.268 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:11.268 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:11.268 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:11.268 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:11.268 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:11.268 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:11.268 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:11.268 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:11.268 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.286+0000 7f0b52ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:11.272 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.287+0000 7f0b52ffd640 10 client.?.objecter ms_handle_connect 0x7f0b640b4100 2024-03-29T10:59:11.272 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.287+0000 7f0b52ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:11.272 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.287+0000 7f0b52ffd640 1 -- 172.21.15.112:0/3930335576 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1627825472 0 0) 0x7f0b64141570 con 0x7f0b640b4100 2024-03-29T10:59:11.272 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.287+0000 7f0b52ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:59:11.272 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.287+0000 7f0b52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:11.272 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.287+0000 7f0b52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:11.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.287+0000 7f0b52ffd640 10 monclient(hunting): my global_id is 4538 2024-03-29T10:59:11.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.288+0000 7f0b52ffd640 1 -- 172.21.15.112:0/3930335576 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b44004eb0 con 0x7f0b640b4100 2024-03-29T10:59:11.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.288+0000 7f0b52ffd640 1 -- 172.21.15.112:0/3930335576 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3038464648 0 0) 0x7f0b64150b70 con 0x7f0b640a9ac0 2024-03-29T10:59:11.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.288+0000 7f0b52ffd640 1 -- 172.21.15.112:0/3930335576 >> v1:172.21.15.112:6790/0 conn(0x7f0b640b4100 legacy=0x7f0b6414fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:11.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.288+0000 7f0b52ffd640 1 -- 172.21.15.112:0/3930335576 >> v1:172.21.15.118:6789/0 conn(0x7f0b6413ee90 legacy=0x7f0b64150460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:11.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.289+0000 7f0b52ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:59:11.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.289+0000 7f0b52ffd640 1 monclient: found mon.a 2024-03-29T10:59:11.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.289+0000 7f0b52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:11.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.289+0000 7f0b52ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:59:11.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.289+0000 7f0b52ffd640 1 -- 172.21.15.112:0/3930335576 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6412eec0 con 0x7f0b640a9ac0 2024-03-29T10:59:11.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.289+0000 7f0b52ffd640 10 monclient: _finish_auth 0 2024-03-29T10:59:11.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.289+0000 7f0b52ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:59:11.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.289+0000 7f0b52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:11.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.289+0000 7f0b52ffd640 1 -- 172.21.15.112:0/3930335576 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b5c001990 con 0x7f0b640a9ac0 2024-03-29T10:59:11.273 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.289+0000 7f0b68ac8640 5 monclient: authenticate success, global_id 4509 2024-03-29T10:59:11.274 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.289+0000 7f0b68ac8640 10 monclient: _renew_subs 2024-03-29T10:59:11.274 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.289+0000 7f0b68ac8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:59:11.274 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.289+0000 7f0b68ac8640 1 -- 172.21.15.112:0/3930335576 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b641334e0 con 0x7f0b640a9ac0 2024-03-29T10:59:11.274 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.289+0000 7f0b68ac8640 10 client.4509.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:11.274 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.289+0000 7f0b68ac8640 10 monclient: _renew_subs 2024-03-29T10:59:11.274 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.289+0000 7f0b68ac8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:59:11.274 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.289+0000 7f0b68ac8640 1 -- 172.21.15.112:0/3930335576 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b6413df90 con 0x7f0b640a9ac0 2024-03-29T10:59:11.274 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.289+0000 7f0b68ac8640 1 librados: init done 2024-03-29T10:59:11.274 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.290+0000 7f0b52ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:11.274 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.290+0000 7f0b52ffd640 1 -- 172.21.15.112:0/3930335576 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f0b5c005250 con 0x7f0b640a9ac0 2024-03-29T10:59:11.274 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.290+0000 7f0b52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:11.274 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.290+0000 7f0b52ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:11.274 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.290+0000 7f0b52ffd640 10 monclient: dump: 2024-03-29T10:59:11.275 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:11.275 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:11.275 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:11.275 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:11.275 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:11.275 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:11.275 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:11.275 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:11.275 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:11.275 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:11.275 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.290+0000 7f0b60d99640 4 set_mon_vals no callback set 2024-03-29T10:59:11.275 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.291+0000 7f0b68ac8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:11.275 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.291+0000 7f0b68ac8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:11.275 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.291+0000 7f0b68ac8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:59:11.275 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.291+0000 7f0b68ac8640 1 -- 172.21.15.112:0/3930335576 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b48005080 con 0x7f0b640a9ac0 2024-03-29T10:59:11.276 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.293+0000 7f0b52ffd640 1 -- 172.21.15.112:0/3930335576 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f0b5c01b220 con 0x7f0b640a9ac0 2024-03-29T10:59:11.276 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.293+0000 7f0b52ffd640 1 -- 172.21.15.112:0/3930335576 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f0b5c084fe0 con 0x7f0b640a9ac0 2024-03-29T10:59:11.277 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.293+0000 7f0b52ffd640 10 client.4509.objecter ms_dispatch 0x7f0b64000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:11.277 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.293+0000 7f0b52ffd640 3 client.4509.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:11.277 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.293+0000 7f0b52ffd640 3 client.4509.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:11.277 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.294+0000 7f0b52ffd640 20 client.4509.objecter dump_active .. 0 homeless 2024-03-29T10:59:11.277 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.294+0000 7f0b61d9b640 10 monclient: get_auth_request con 0x7f0b4406dde0 auth_method 0 2024-03-29T10:59:11.277 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.294+0000 7f0b52ffd640 10 client.4509.objecter ms_handle_connect 0x7f0b4406dde0 2024-03-29T10:59:11.280 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.299+0000 7f0b52ffd640 1 -- 172.21.15.112:0/3930335576 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b5c0508c0 con 0x7f0b640a9ac0 2024-03-29T10:59:11.280 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.300+0000 7f0b52ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:11.280 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.300+0000 7f0b52ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:11.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.629+0000 7f0b68ac8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:11.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.629+0000 7f0b68ac8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:11.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.629+0000 7f0b68ac8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:59:11.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.629+0000 7f0b68ac8640 1 -- 172.21.15.112:0/3930335576 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b48005330 con 0x7f0b640a9ac0 2024-03-29T10:59:11.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.632+0000 7f0b52ffd640 1 -- 172.21.15.112:0/3930335576 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f0b5c0527f0 con 0x7f0b640a9ac0 2024-03-29T10:59:11.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.632+0000 7f0b52ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:11.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.632+0000 7f0b52ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:11.613 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:59:11.613 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:11.616 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.636+0000 7f0b68ac8640 10 librados: watch_flush enter 2024-03-29T10:59:11.616 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.636+0000 7f0b68ac8640 10 librados: watch_flush exit 2024-03-29T10:59:11.616 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.636+0000 7f0b68ac8640 20 client.4509.objecter shutdown clearing up homeless session... 2024-03-29T10:59:11.616 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.636+0000 7f0b68ac8640 10 client.4509.objecter successfully canceled tick 2024-03-29T10:59:11.616 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.636+0000 7f0b68ac8640 1 -- 172.21.15.112:0/3930335576 >> v1:172.21.15.118:6816/3181073408 conn(0x7f0b4406dde0 legacy=0x7f0b440702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:11.617 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.636+0000 7f0b68ac8640 10 monclient: shutdown 2024-03-29T10:59:11.617 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.636+0000 7f0b68ac8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:11.617 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.636+0000 7f0b68ac8640 1 -- 172.21.15.112:0/3930335576 >> v1:172.21.15.112:6789/0 conn(0x7f0b640a9ac0 legacy=0x7f0b6413f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:11.617 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.637+0000 7f0b68ac8640 1 -- 172.21.15.112:0/3930335576 shutdown_connections 2024-03-29T10:59:11.617 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.637+0000 7f0b68ac8640 1 -- 172.21.15.112:0/3930335576 >> 172.21.15.112:0/3930335576 conn(0x7f0b640b35e0 msgr2=0x7f0b640b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:11.618 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.637+0000 7f0b68ac8640 1 -- 172.21.15.112:0/3930335576 shutdown_connections 2024-03-29T10:59:11.618 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.637+0000 7f0b68ac8640 1 -- 172.21.15.112:0/3930335576 wait complete. 2024-03-29T10:59:11.618 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:11.637+0000 7f0b68ac8640 1 librados: shutdown 2024-03-29T10:59:11.634 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:18.441 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:18.589 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.608+0000 7f4ff1231640 10 monclient: get_monmap_and_config 2024-03-29T10:59:18.589 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.608+0000 7f4ff1231640 10 monclient: build_initial_monmap 2024-03-29T10:59:18.589 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.608+0000 7f4ff1231640 10 monclient: monmap: 2024-03-29T10:59:18.589 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:59:18.589 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:18.589 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:59:18.609855+0000 2024-03-29T10:59:18.589 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:59:18.609855+0000 2024-03-29T10:59:18.589 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:18.589 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:18.590 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:59:18.590 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:59:18.590 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:59:18.590 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:18.590 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.609+0000 7f4ff1231640 1 Processor -- start 2024-03-29T10:59:18.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.610+0000 7f4ff1231640 1 -- start start 2024-03-29T10:59:18.591 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.610+0000 7f4ff1231640 10 monclient: init 2024-03-29T10:59:18.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.611+0000 7f4ff1231640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:18.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.611+0000 7f4ff1231640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:59:18.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.611+0000 7f4ff1231640 10 monclient(hunting): picked mon.noname-a con 0x7f4fec1457f0 addr v1:172.21.15.112:6789/0 2024-03-29T10:59:18.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.611+0000 7f4ff1231640 10 monclient(hunting): picked mon.noname-b con 0x7f4fec13f650 addr v1:172.21.15.112:6790/0 2024-03-29T10:59:18.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.611+0000 7f4ff1231640 10 monclient(hunting): picked mon.noname-c con 0x7f4fec0a9d70 addr v1:172.21.15.118:6789/0 2024-03-29T10:59:18.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.611+0000 7f4ff1231640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fec0aaef0 con 0x7f4fec1457f0 2024-03-29T10:59:18.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.611+0000 7f4ff1231640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fec145bd0 con 0x7f4fec0a9d70 2024-03-29T10:59:18.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.611+0000 7f4ff1231640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fec145eb0 con 0x7f4fec13f650 2024-03-29T10:59:18.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.611+0000 7f4ff1231640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:18.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.611+0000 7f4ff1231640 10 monclient(hunting): authenticate will time out at 1313.915947s 2024-03-29T10:59:18.594 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.613+0000 7f4fea59c640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f4fec13f650 0x7f4fec0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:58530/0 (socket says 172.21.15.112:58530) 2024-03-29T10:59:18.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.613+0000 7f4fea59c640 1 -- 172.21.15.112:0/489545723 learned_addr learned my addr 172.21.15.112:0/489545723 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:59:18.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.615+0000 7f4fe9d9b640 1 -- 172.21.15.112:0/489545723 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434005156 0 0) 0x7f4fec0aaef0 con 0x7f4fec1457f0 2024-03-29T10:59:18.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.615+0000 7f4fe9d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:59:18.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.615+0000 7f4fe9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:18.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.615+0000 7f4fe9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:18.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.615+0000 7f4fe9d9b640 10 monclient(hunting): my global_id is 4512 2024-03-29T10:59:18.595 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.615+0000 7f4fe9d9b640 1 -- 172.21.15.112:0/489545723 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fd8003520 con 0x7f4fec1457f0 2024-03-29T10:59:18.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.615+0000 7f4fe9d9b640 1 -- 172.21.15.112:0/489545723 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2372119704 0 0) 0x7f4fd8003520 con 0x7f4fec1457f0 2024-03-29T10:59:18.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.615+0000 7f4fe9d9b640 1 -- 172.21.15.112:0/489545723 >> v1:172.21.15.112:6790/0 conn(0x7f4fec13f650 legacy=0x7f4fec0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:18.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.615+0000 7f4fe9d9b640 1 -- 172.21.15.112:0/489545723 >> v1:172.21.15.118:6789/0 conn(0x7f4fec0a9d70 legacy=0x7f4fec14d8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:59:18.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.615+0000 7f4fe9d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:59:18.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.615+0000 7f4fe9d9b640 1 monclient: found mon.noname-a 2024-03-29T10:59:18.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.615+0000 7f4fe9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:18.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.615+0000 7f4fe9d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T10:59:18.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.615+0000 7f4fe9d9b640 1 -- 172.21.15.112:0/489545723 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fec146340 con 0x7f4fec1457f0 2024-03-29T10:59:18.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.615+0000 7f4fe9d9b640 10 monclient: _finish_auth 0 2024-03-29T10:59:18.596 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.615+0000 7f4fe9d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:59:18.597 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.615+0000 7f4fe9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:18.597 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.615+0000 7f4fe9d9b640 1 -- 172.21.15.112:0/489545723 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f4fd4001690 con 0x7f4fec1457f0 2024-03-29T10:59:18.597 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.615+0000 7f4fe9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:18.597 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.615+0000 7f4fe9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:59:18.597 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.615+0000 7f4fe9d9b640 10 monclient: dump: 2024-03-29T10:59:18.597 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:18.597 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:18.597 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:18.597 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:18.597 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:18.597 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:18.597 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:18.597 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:18.597 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:18.597 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:18.597 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.616+0000 7f4fe9d9b640 1 -- 172.21.15.112:0/489545723 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fd4001860 con 0x7f4fec1457f0 2024-03-29T10:59:18.598 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.616+0000 7f4fe9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:18.598 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.616+0000 7f4fe9d9b640 1 -- 172.21.15.112:0/489545723 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f4fd4001cb0 con 0x7f4fec1457f0 2024-03-29T10:59:18.598 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.616+0000 7f4fe9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:18.598 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.616+0000 7f4fe9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:18.598 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.616+0000 7f4fe9d9b640 10 monclient: dump: 2024-03-29T10:59:18.598 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:18.598 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:18.598 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:18.598 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:18.598 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:18.598 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:18.598 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:18.598 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:18.598 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:18.598 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:18.598 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.616+0000 7f4ff1231640 5 monclient: authenticate success, global_id 4512 2024-03-29T10:59:18.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.616+0000 7f4ff1231640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:18.599 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.616+0000 7f4ff1231640 4 set_mon_vals no callback set 2024-03-29T10:59:18.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.617+0000 7f4ff1231640 10 monclient: shutdown 2024-03-29T10:59:18.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.617+0000 7f4ff1231640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:18.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.617+0000 7f4ff1231640 1 -- 172.21.15.112:0/489545723 >> v1:172.21.15.112:6789/0 conn(0x7f4fec1457f0 legacy=0x7f4fec13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:18.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.617+0000 7f4ff1231640 1 -- 172.21.15.112:0/489545723 shutdown_connections 2024-03-29T10:59:18.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.617+0000 7f4ff1231640 1 -- 172.21.15.112:0/489545723 >> 172.21.15.112:0/489545723 conn(0x7f4fec066a00 msgr2=0x7f4fec066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:18.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.617+0000 7f4ff1231640 1 -- 172.21.15.112:0/489545723 shutdown_connections 2024-03-29T10:59:18.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.617+0000 7f4ff1231640 1 -- 172.21.15.112:0/489545723 wait complete. 2024-03-29T10:59:18.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.618+0000 7f4ff1231640 10 monclient: build_initial_monmap 2024-03-29T10:59:18.601 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.618+0000 7f4ff1231640 10 monclient: monmap: 2024-03-29T10:59:18.601 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:59:18.601 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:18.602 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:59:18.602 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:59:18.602 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:18.602 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:18.602 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:59:18.602 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:59:18.602 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:59:18.602 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:18.602 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.619+0000 7f4ff1231640 1 librados: starting msgr at 2024-03-29T10:59:18.602 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.619+0000 7f4ff1231640 1 librados: starting objecter 2024-03-29T10:59:18.602 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.619+0000 7f4ff1231640 1 Processor -- start 2024-03-29T10:59:18.602 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.619+0000 7f4ff1231640 1 -- start start 2024-03-29T10:59:18.602 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.619+0000 7f4ff1231640 1 librados: setting wanted keys 2024-03-29T10:59:18.602 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.619+0000 7f4ff1231640 1 librados: calling monclient init 2024-03-29T10:59:18.602 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.619+0000 7f4ff1231640 10 monclient: init 2024-03-29T10:59:18.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.620+0000 7f4ff1231640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:18.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.620+0000 7f4ff1231640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:59:18.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.620+0000 7f4ff1231640 10 monclient(hunting): picked mon.noname-b con 0x7f4fec0a9d70 addr v1:172.21.15.118:6789/0 2024-03-29T10:59:18.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.620+0000 7f4ff1231640 10 monclient(hunting): picked mon.noname-c con 0x7f4fec13f650 addr v1:172.21.15.112:6790/0 2024-03-29T10:59:18.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.620+0000 7f4ff1231640 10 monclient(hunting): picked mon.noname-a con 0x7f4fec1457f0 addr v1:172.21.15.112:6789/0 2024-03-29T10:59:18.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.620+0000 7f4ff1231640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fec06dfa0 con 0x7f4fec1457f0 2024-03-29T10:59:18.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.620+0000 7f4ff1231640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fec071fc0 con 0x7f4fec0a9d70 2024-03-29T10:59:18.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.620+0000 7f4ff1231640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fec1df800 con 0x7f4fec13f650 2024-03-29T10:59:18.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.620+0000 7f4ff1231640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:18.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.620+0000 7f4ff1231640 10 monclient(hunting): authenticate will time out at 1313.924782s 2024-03-29T10:59:18.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.620+0000 7f4fea59c640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f4fec13f650 0x7f4fec1410a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59310/0 (socket says 172.21.15.112:59310) 2024-03-29T10:59:18.603 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.621+0000 7f4fea59c640 1 -- 172.21.15.112:0/4200837911 learned_addr learned my addr 172.21.15.112:0/4200837911 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:59:18.608 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.622+0000 7f4fe37fe640 10 client.?.objecter ms_handle_connect 0x7f4fec13f650 2024-03-29T10:59:18.608 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.622+0000 7f4fe37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:18.608 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.623+0000 7f4fe37fe640 1 -- 172.21.15.112:0/4200837911 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540523094 0 0) 0x7f4fec1df800 con 0x7f4fec13f650 2024-03-29T10:59:18.608 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.623+0000 7f4fe37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:59:18.608 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.623+0000 7f4fe37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:18.608 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.623+0000 7f4fe37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:18.608 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.623+0000 7f4fe37fe640 10 monclient(hunting): my global_id is 4544 2024-03-29T10:59:18.608 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.623+0000 7f4fe37fe640 1 -- 172.21.15.112:0/4200837911 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fcc004eb0 con 0x7f4fec13f650 2024-03-29T10:59:18.608 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.623+0000 7f4fe37fe640 10 client.?.objecter ms_handle_connect 0x7f4fec0a9d70 2024-03-29T10:59:18.608 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.623+0000 7f4fe37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:18.608 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.623+0000 7f4fe37fe640 1 -- 172.21.15.112:0/4200837911 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424606205 0 0) 0x7f4fec071fc0 con 0x7f4fec0a9d70 2024-03-29T10:59:18.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.623+0000 7f4fe37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:59:18.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.623+0000 7f4fe37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:18.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.623+0000 7f4fe37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:18.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.623+0000 7f4fe37fe640 10 monclient(hunting): my global_id is 4489 2024-03-29T10:59:18.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.623+0000 7f4fe37fe640 1 -- 172.21.15.112:0/4200837911 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fcc0051d0 con 0x7f4fec0a9d70 2024-03-29T10:59:18.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.624+0000 7f4fe37fe640 1 -- 172.21.15.112:0/4200837911 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3462579714 0 0) 0x7f4fcc004eb0 con 0x7f4fec13f650 2024-03-29T10:59:18.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.624+0000 7f4fe37fe640 1 -- 172.21.15.112:0/4200837911 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fec071fc0 con 0x7f4fec13f650 2024-03-29T10:59:18.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.624+0000 7f4fe37fe640 1 -- 172.21.15.112:0/4200837911 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f4fdc001930 con 0x7f4fec13f650 2024-03-29T10:59:18.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.624+0000 7f4fe37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:18.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.624+0000 7f4fe37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:18.609 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.624+0000 7f4fe37fe640 10 monclient(hunting): dump: 2024-03-29T10:59:18.609 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:18.609 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:18.609 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:18.609 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:18.610 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:18.610 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:18.610 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:18.610 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:18.610 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:18.610 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:18.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.624+0000 7f4fe37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:18.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.624+0000 7f4fe37fe640 1 -- 172.21.15.112:0/4200837911 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1562916572 0 0) 0x7f4fcc0051d0 con 0x7f4fec0a9d70 2024-03-29T10:59:18.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.624+0000 7f4fe37fe640 1 -- 172.21.15.112:0/4200837911 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fcc004eb0 con 0x7f4fec0a9d70 2024-03-29T10:59:18.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.624+0000 7f4fe37fe640 1 -- 172.21.15.112:0/4200837911 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f4fd4001b70 con 0x7f4fec0a9d70 2024-03-29T10:59:18.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.624+0000 7f4fe37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:18.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.624+0000 7f4fe37fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:18.610 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.624+0000 7f4fe37fe640 10 monclient(hunting): dump: 2024-03-29T10:59:18.610 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:18.610 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:18.611 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:18.611 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:18.611 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:18.611 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:18.611 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:18.611 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:18.611 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:18.611 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:18.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.624+0000 7f4fe37fe640 1 -- 172.21.15.112:0/4200837911 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 353708300 0 0) 0x7f4fcc004eb0 con 0x7f4fec0a9d70 2024-03-29T10:59:18.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.624+0000 7f4fe37fe640 1 -- 172.21.15.112:0/4200837911 >> v1:172.21.15.112:6790/0 conn(0x7f4fec13f650 legacy=0x7f4fec1410a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:18.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.624+0000 7f4fe37fe640 1 -- 172.21.15.112:0/4200837911 >> v1:172.21.15.112:6789/0 conn(0x7f4fec1457f0 legacy=0x7f4fec1417b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:59:18.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.626+0000 7f4fe37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:59:18.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.626+0000 7f4fe37fe640 1 monclient: found mon.b 2024-03-29T10:59:18.611 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.626+0000 7f4fe37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:18.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.626+0000 7f4fe37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:59:18.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.626+0000 7f4fe37fe640 1 -- 172.21.15.112:0/4200837911 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fec1df990 con 0x7f4fec0a9d70 2024-03-29T10:59:18.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.626+0000 7f4fe37fe640 10 monclient: _finish_auth 0 2024-03-29T10:59:18.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.626+0000 7f4fe37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:59:18.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.626+0000 7f4fe37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:18.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.626+0000 7f4ff1231640 5 monclient: authenticate success, global_id 4489 2024-03-29T10:59:18.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.626+0000 7f4ff1231640 10 monclient: _renew_subs 2024-03-29T10:59:18.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.626+0000 7f4ff1231640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:59:18.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.626+0000 7f4ff1231640 1 -- 172.21.15.112:0/4200837911 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fec1dfbe0 con 0x7f4fec0a9d70 2024-03-29T10:59:18.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.626+0000 7f4fe37fe640 1 -- 172.21.15.112:0/4200837911 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fd40038a0 con 0x7f4fec0a9d70 2024-03-29T10:59:18.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.626+0000 7f4fe37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:18.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.626+0000 7f4fe37fe640 1 -- 172.21.15.112:0/4200837911 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f4fd4003bb0 con 0x7f4fec0a9d70 2024-03-29T10:59:18.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.626+0000 7f4fe37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:18.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.626+0000 7f4fe37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:18.612 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.626+0000 7f4fe37fe640 10 monclient: dump: 2024-03-29T10:59:18.613 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:18.613 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:18.613 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:18.613 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:18.613 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:18.613 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:18.613 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:18.613 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:18.613 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:18.613 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:18.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.626+0000 7f4ff1231640 10 client.4489.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:18.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.626+0000 7f4ff1231640 10 monclient: _renew_subs 2024-03-29T10:59:18.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.626+0000 7f4ff1231640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:59:18.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.626+0000 7f4ff1231640 1 -- 172.21.15.112:0/4200837911 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fec1e75b0 con 0x7f4fec0a9d70 2024-03-29T10:59:18.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.626+0000 7f4ff1231640 1 librados: init done 2024-03-29T10:59:18.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.626+0000 7f4fe8d99640 4 set_mon_vals no callback set 2024-03-29T10:59:18.613 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.627+0000 7f4ff1231640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:18.615 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.627+0000 7f4ff1231640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:18.615 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.627+0000 7f4ff1231640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:59:18.615 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.627+0000 7f4ff1231640 1 -- 172.21.15.112:0/4200837911 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fd0005080 con 0x7f4fec0a9d70 2024-03-29T10:59:18.615 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.628+0000 7f4fe37fe640 1 -- 172.21.15.112:0/4200837911 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f4fd4019b80 con 0x7f4fec0a9d70 2024-03-29T10:59:18.615 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.629+0000 7f4fe37fe640 1 -- 172.21.15.112:0/4200837911 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f4fd40536a0 con 0x7f4fec0a9d70 2024-03-29T10:59:18.615 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.629+0000 7f4fe37fe640 10 client.4489.objecter ms_dispatch 0x7f4fec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:18.615 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.629+0000 7f4fe37fe640 3 client.4489.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:18.615 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.629+0000 7f4fe37fe640 3 client.4489.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:18.615 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.629+0000 7f4fe37fe640 20 client.4489.objecter dump_active .. 0 homeless 2024-03-29T10:59:18.615 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.629+0000 7f4fea59c640 10 monclient: get_auth_request con 0x7f4fcc06df50 auth_method 0 2024-03-29T10:59:18.615 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.629+0000 7f4fe37fe640 10 client.4489.objecter ms_handle_connect 0x7f4fcc06df50 2024-03-29T10:59:18.618 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.638+0000 7f4fe37fe640 1 -- 172.21.15.112:0/4200837911 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4fd404f940 con 0x7f4fec0a9d70 2024-03-29T10:59:18.618 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.638+0000 7f4fe37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:18.618 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.638+0000 7f4fe37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:18.944 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.963+0000 7f4ff1231640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:18.944 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.963+0000 7f4ff1231640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:18.944 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.963+0000 7f4ff1231640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:59:18.944 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.963+0000 7f4ff1231640 1 -- 172.21.15.112:0/4200837911 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4fd0005330 con 0x7f4fec0a9d70 2024-03-29T10:59:18.946 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.965+0000 7f4fe37fe640 1 -- 172.21.15.112:0/4200837911 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f4fd404fae0 con 0x7f4fec0a9d70 2024-03-29T10:59:18.946 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.965+0000 7f4fe37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:18.946 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.965+0000 7f4fe37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:18.947 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:59:18.947 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:18.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.970+0000 7f4ff1231640 10 librados: watch_flush enter 2024-03-29T10:59:18.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.970+0000 7f4ff1231640 10 librados: watch_flush exit 2024-03-29T10:59:18.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.970+0000 7f4ff1231640 20 client.4489.objecter shutdown clearing up homeless session... 2024-03-29T10:59:18.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.970+0000 7f4ff1231640 10 client.4489.objecter successfully canceled tick 2024-03-29T10:59:18.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.970+0000 7f4ff1231640 1 -- 172.21.15.112:0/4200837911 >> v1:172.21.15.118:6816/3181073408 conn(0x7f4fcc06df50 legacy=0x7f4fcc070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:18.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.970+0000 7f4ff1231640 10 monclient: shutdown 2024-03-29T10:59:18.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.970+0000 7f4ff1231640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:18.951 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.970+0000 7f4ff1231640 1 -- 172.21.15.112:0/4200837911 >> v1:172.21.15.118:6789/0 conn(0x7f4fec0a9d70 legacy=0x7f4fec1df0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:18.951 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.970+0000 7f4ff1231640 1 -- 172.21.15.112:0/4200837911 shutdown_connections 2024-03-29T10:59:18.951 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.971+0000 7f4ff1231640 1 -- 172.21.15.112:0/4200837911 >> 172.21.15.112:0/4200837911 conn(0x7f4fec066a00 msgr2=0x7f4fec0b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:18.951 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.971+0000 7f4ff1231640 1 -- 172.21.15.112:0/4200837911 shutdown_connections 2024-03-29T10:59:18.952 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.971+0000 7f4ff1231640 1 -- 172.21.15.112:0/4200837911 wait complete. 2024-03-29T10:59:18.952 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:18.971+0000 7f4ff1231640 1 librados: shutdown 2024-03-29T10:59:18.968 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:25.775 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:25.926 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.945+0000 7f977c704640 10 monclient: get_monmap_and_config 2024-03-29T10:59:25.927 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.945+0000 7f977c704640 10 monclient: build_initial_monmap 2024-03-29T10:59:25.927 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.945+0000 7f977c704640 10 monclient: monmap: 2024-03-29T10:59:25.927 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:59:25.927 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:25.927 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:59:25.946242+0000 2024-03-29T10:59:25.927 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:59:25.946242+0000 2024-03-29T10:59:25.927 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:25.927 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:25.927 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:59:25.927 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:59:25.927 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:59:25.928 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:25.928 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.946+0000 7f977c704640 1 Processor -- start 2024-03-29T10:59:25.928 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.946+0000 7f977c704640 1 -- start start 2024-03-29T10:59:25.928 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.946+0000 7f977c704640 10 monclient: init 2024-03-29T10:59:25.930 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.947+0000 7f977c704640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:25.930 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.947+0000 7f977c704640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:59:25.930 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.947+0000 7f977c704640 10 monclient(hunting): picked mon.noname-b con 0x7f97740a94e0 addr v1:172.21.15.112:6790/0 2024-03-29T10:59:25.930 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.947+0000 7f977c704640 10 monclient(hunting): picked mon.noname-c con 0x7f977413b5a0 addr v1:172.21.15.118:6789/0 2024-03-29T10:59:25.930 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.947+0000 7f977c704640 10 monclient(hunting): picked mon.noname-a con 0x7f97740a9ba0 addr v1:172.21.15.112:6789/0 2024-03-29T10:59:25.930 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.947+0000 7f977c704640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97740ac430 con 0x7f97740a9ba0 2024-03-29T10:59:25.931 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.947+0000 7f977c704640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97740ac680 con 0x7f977413b5a0 2024-03-29T10:59:25.931 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.947+0000 7f977c704640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97740ac960 con 0x7f97740a94e0 2024-03-29T10:59:25.931 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.947+0000 7f977c704640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:25.931 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.947+0000 7f977c704640 10 monclient(hunting): authenticate will time out at 1321.251457s 2024-03-29T10:59:25.931 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.948+0000 7f977a4a0640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f97740a94e0 0x7f977413ae90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35366/0 (socket says 172.21.15.112:35366) 2024-03-29T10:59:25.931 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.948+0000 7f977a4a0640 1 -- 172.21.15.112:0/2123649133 learned_addr learned my addr 172.21.15.112:0/2123649133 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:59:25.931 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.949+0000 7f977949e640 1 -- 172.21.15.112:0/2123649133 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354314627 0 0) 0x7f97740ac960 con 0x7f97740a94e0 2024-03-29T10:59:25.931 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.949+0000 7f977949e640 10 monclient(hunting): _negotiate 2024-03-29T10:59:25.931 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.949+0000 7f977949e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:25.931 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.949+0000 7f977949e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:25.931 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.949+0000 7f977949e640 10 monclient(hunting): my global_id is 4547 2024-03-29T10:59:25.931 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.949+0000 7f977949e640 1 -- 172.21.15.112:0/2123649133 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9760003520 con 0x7f97740a94e0 2024-03-29T10:59:25.931 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.949+0000 7f977949e640 1 -- 172.21.15.112:0/2123649133 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2634173932 0 0) 0x7f9760003520 con 0x7f97740a94e0 2024-03-29T10:59:25.931 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.949+0000 7f977949e640 1 -- 172.21.15.112:0/2123649133 >> v1:172.21.15.118:6789/0 conn(0x7f977413b5a0 legacy=0x7f97740aae40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:59:25.931 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.950+0000 7f977949e640 1 -- 172.21.15.112:0/2123649133 >> v1:172.21.15.112:6789/0 conn(0x7f97740a9ba0 legacy=0x7f9774142890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:59:25.934 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.952+0000 7f977949e640 10 monclient: _finish_hunting 0 2024-03-29T10:59:25.934 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.952+0000 7f977949e640 1 monclient: found mon.noname-b 2024-03-29T10:59:25.934 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.952+0000 7f977949e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:25.934 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.952+0000 7f977949e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:59:25.934 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.952+0000 7f977949e640 1 -- 172.21.15.112:0/2123649133 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f977413baf0 con 0x7f97740a94e0 2024-03-29T10:59:25.934 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.952+0000 7f977949e640 10 monclient: _finish_auth 0 2024-03-29T10:59:25.934 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.952+0000 7f977949e640 10 monclient: _check_auth_tickets 2024-03-29T10:59:25.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.952+0000 7f977949e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:25.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.952+0000 7f977c704640 5 monclient: authenticate success, global_id 4547 2024-03-29T10:59:25.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.952+0000 7f977c704640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:25.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.952+0000 7f977949e640 1 -- 172.21.15.112:0/2123649133 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9764001690 con 0x7f97740a94e0 2024-03-29T10:59:25.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.952+0000 7f977949e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:25.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.952+0000 7f977949e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:25.935 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.952+0000 7f977949e640 10 monclient: dump: 2024-03-29T10:59:25.935 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:25.935 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:25.935 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:25.935 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:25.935 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:25.935 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:25.935 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:25.936 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:25.936 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:25.936 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:25.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.952+0000 7f977c704640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:25.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.952+0000 7f977949e640 1 -- 172.21.15.112:0/2123649133 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9764001920 con 0x7f97740a94e0 2024-03-29T10:59:25.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.953+0000 7f977949e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:25.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.953+0000 7f977c704640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:25.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.953+0000 7f977c704640 4 set_mon_vals no callback set 2024-03-29T10:59:25.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.953+0000 7f977949e640 1 -- 172.21.15.112:0/2123649133 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9764001cb0 con 0x7f97740a94e0 2024-03-29T10:59:25.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.953+0000 7f977949e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:25.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.953+0000 7f977949e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:25.936 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.953+0000 7f977949e640 10 monclient: dump: 2024-03-29T10:59:25.936 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:25.936 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:25.936 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:25.937 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:25.937 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:25.937 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:25.937 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:25.937 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:25.937 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:25.937 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:25.937 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.954+0000 7f977c704640 10 monclient: shutdown 2024-03-29T10:59:25.937 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.954+0000 7f977c704640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:25.937 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.954+0000 7f977c704640 1 -- 172.21.15.112:0/2123649133 >> v1:172.21.15.112:6790/0 conn(0x7f97740a94e0 legacy=0x7f977413ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:25.937 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.954+0000 7f977c704640 1 -- 172.21.15.112:0/2123649133 shutdown_connections 2024-03-29T10:59:25.937 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.954+0000 7f977c704640 1 -- 172.21.15.112:0/2123649133 >> 172.21.15.112:0/2123649133 conn(0x7f97740b3a00 msgr2=0x7f97740b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:25.937 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.954+0000 7f977c704640 1 -- 172.21.15.112:0/2123649133 shutdown_connections 2024-03-29T10:59:25.937 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.954+0000 7f977c704640 1 -- 172.21.15.112:0/2123649133 wait complete. 2024-03-29T10:59:25.941 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.955+0000 7f977c704640 10 monclient: build_initial_monmap 2024-03-29T10:59:25.941 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.955+0000 7f977c704640 10 monclient: monmap: 2024-03-29T10:59:25.941 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:59:25.941 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:25.941 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:59:25.941 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:59:25.941 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:25.941 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:25.942 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:59:25.942 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:59:25.942 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:59:25.942 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:25.942 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.955+0000 7f977c704640 1 librados: starting msgr at 2024-03-29T10:59:25.942 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.955+0000 7f977c704640 1 librados: starting objecter 2024-03-29T10:59:25.942 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.955+0000 7f977c704640 1 Processor -- start 2024-03-29T10:59:25.942 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.955+0000 7f977c704640 1 -- start start 2024-03-29T10:59:25.942 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.955+0000 7f977c704640 1 librados: setting wanted keys 2024-03-29T10:59:25.942 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.955+0000 7f977c704640 1 librados: calling monclient init 2024-03-29T10:59:25.942 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.955+0000 7f977c704640 10 monclient: init 2024-03-29T10:59:25.942 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.956+0000 7f977c704640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:25.942 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.956+0000 7f977c704640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:59:25.942 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.956+0000 7f977c704640 10 monclient(hunting): picked mon.noname-c con 0x7f97740a94e0 addr v1:172.21.15.112:6790/0 2024-03-29T10:59:25.943 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.956+0000 7f977c704640 10 monclient(hunting): picked mon.noname-b con 0x7f97740a9ba0 addr v1:172.21.15.118:6789/0 2024-03-29T10:59:25.943 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.956+0000 7f977c704640 10 monclient(hunting): picked mon.noname-a con 0x7f977413b5a0 addr v1:172.21.15.112:6789/0 2024-03-29T10:59:25.943 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.956+0000 7f977c704640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977406f750 con 0x7f977413b5a0 2024-03-29T10:59:25.943 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.956+0000 7f977c704640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9774071c10 con 0x7f97740a9ba0 2024-03-29T10:59:25.943 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.956+0000 7f977c704640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9774072bb0 con 0x7f97740a94e0 2024-03-29T10:59:25.943 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.956+0000 7f977c704640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:25.943 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.956+0000 7f977c704640 10 monclient(hunting): authenticate will time out at 1321.260716s 2024-03-29T10:59:25.943 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.958+0000 7f977a4a0640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f97740a94e0 0x7f977413e320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35896/0 (socket says 172.21.15.112:35896) 2024-03-29T10:59:25.943 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.958+0000 7f977a4a0640 1 -- 172.21.15.112:0/1341221446 learned_addr learned my addr 172.21.15.112:0/1341221446 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:59:25.943 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.958+0000 7f976affd640 10 client.?.objecter ms_handle_connect 0x7f97740a94e0 2024-03-29T10:59:25.943 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.958+0000 7f976affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:25.943 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.958+0000 7f976affd640 1 -- 172.21.15.112:0/1341221446 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584138554 0 0) 0x7f9774072bb0 con 0x7f97740a94e0 2024-03-29T10:59:25.943 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.958+0000 7f976affd640 10 monclient(hunting): _negotiate 2024-03-29T10:59:25.943 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.958+0000 7f976affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:25.944 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.958+0000 7f976affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:25.944 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.958+0000 7f976affd640 10 monclient(hunting): my global_id is 4550 2024-03-29T10:59:25.944 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.958+0000 7f976affd640 1 -- 172.21.15.112:0/1341221446 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9758004eb0 con 0x7f97740a94e0 2024-03-29T10:59:25.944 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.960+0000 7f976affd640 1 -- 172.21.15.112:0/1341221446 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3821760933 0 0) 0x7f9758004eb0 con 0x7f97740a94e0 2024-03-29T10:59:25.944 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.960+0000 7f976affd640 1 -- 172.21.15.112:0/1341221446 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9758005300 con 0x7f97740a94e0 2024-03-29T10:59:25.944 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.960+0000 7f976affd640 1 -- 172.21.15.112:0/1341221446 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9764001610 con 0x7f97740a94e0 2024-03-29T10:59:25.944 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.960+0000 7f976affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:25.944 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.960+0000 7f976affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:25.944 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.960+0000 7f976affd640 10 monclient(hunting): dump: 2024-03-29T10:59:25.944 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:25.944 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:25.944 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:25.944 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:25.944 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:25.944 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:25.945 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:25.945 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:25.945 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:25.945 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:25.945 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.960+0000 7f976affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:25.945 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.960+0000 7f976affd640 1 -- 172.21.15.112:0/1341221446 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2460991988 0 0) 0x7f9758005300 con 0x7f97740a94e0 2024-03-29T10:59:25.945 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.960+0000 7f976affd640 1 -- 172.21.15.112:0/1341221446 >> v1:172.21.15.118:6789/0 conn(0x7f97740a9ba0 legacy=0x7f9774070070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:59:25.947 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.961+0000 7f976affd640 1 -- 172.21.15.112:0/1341221446 >> v1:172.21.15.112:6789/0 conn(0x7f977413b5a0 legacy=0x7f97741e1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:25.947 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.961+0000 7f976affd640 10 monclient: _finish_hunting 0 2024-03-29T10:59:25.947 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.961+0000 7f976affd640 1 monclient: found mon.c 2024-03-29T10:59:25.948 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.961+0000 7f976affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:25.948 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.961+0000 7f976affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:59:25.948 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.961+0000 7f976affd640 1 -- 172.21.15.112:0/1341221446 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9774138400 con 0x7f97740a94e0 2024-03-29T10:59:25.948 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.961+0000 7f976affd640 10 monclient: _finish_auth 0 2024-03-29T10:59:25.948 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.961+0000 7f976affd640 10 monclient: _check_auth_tickets 2024-03-29T10:59:25.948 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.961+0000 7f976affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:25.948 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.961+0000 7f976affd640 10 client.?.objecter ms_handle_connect 0x7f977413b5a0 2024-03-29T10:59:25.948 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.961+0000 7f976affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:25.948 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.962+0000 7f977c704640 5 monclient: authenticate success, global_id 4550 2024-03-29T10:59:25.948 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.962+0000 7f977c704640 10 monclient: _renew_subs 2024-03-29T10:59:25.948 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.962+0000 7f977c704640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:59:25.948 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.962+0000 7f977c704640 1 -- 172.21.15.112:0/1341221446 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9774070780 con 0x7f97740a94e0 2024-03-29T10:59:25.948 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.962+0000 7f977c704640 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:25.948 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.962+0000 7f977c704640 10 monclient: _renew_subs 2024-03-29T10:59:25.948 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.962+0000 7f977c704640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:59:25.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.962+0000 7f977c704640 1 -- 172.21.15.112:0/1341221446 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f977406fdc0 con 0x7f97740a94e0 2024-03-29T10:59:25.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.962+0000 7f977c704640 1 librados: init done 2024-03-29T10:59:25.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.963+0000 7f976affd640 1 -- 172.21.15.112:0/1341221446 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9764004bf0 con 0x7f97740a94e0 2024-03-29T10:59:25.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.963+0000 7f976affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:25.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.963+0000 7f976affd640 1 -- 172.21.15.112:0/1341221446 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9764001610 con 0x7f97740a94e0 2024-03-29T10:59:25.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.963+0000 7f976affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:25.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.963+0000 7f976affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:25.949 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.963+0000 7f976affd640 10 monclient: dump: 2024-03-29T10:59:25.949 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:25.949 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:25.949 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:25.949 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:25.949 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:25.949 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:25.949 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:25.950 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:25.950 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:25.950 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:25.950 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.963+0000 7f9778c9d640 4 set_mon_vals no callback set 2024-03-29T10:59:25.951 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.968+0000 7f977c704640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:25.951 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.968+0000 7f977c704640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:25.951 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.968+0000 7f977c704640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:59:25.951 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.968+0000 7f977c704640 1 -- 172.21.15.112:0/1341221446 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f975c005080 con 0x7f97740a94e0 2024-03-29T10:59:25.951 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.969+0000 7f976affd640 1 -- 172.21.15.112:0/1341221446 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f976401abd0 con 0x7f97740a94e0 2024-03-29T10:59:25.951 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.969+0000 7f976affd640 1 -- 172.21.15.112:0/1341221446 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f9764084a00 con 0x7f97740a94e0 2024-03-29T10:59:25.952 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.969+0000 7f976affd640 10 client.4550.objecter ms_dispatch 0x7f9774000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:25.952 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.969+0000 7f976affd640 3 client.4550.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:25.952 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.969+0000 7f976affd640 3 client.4550.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:25.952 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.969+0000 7f976affd640 20 client.4550.objecter dump_active .. 0 homeless 2024-03-29T10:59:25.952 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.970+0000 7f9779c9f640 10 monclient: get_auth_request con 0x7f975806df90 auth_method 0 2024-03-29T10:59:25.956 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.976+0000 7f976affd640 10 client.4550.objecter ms_handle_connect 0x7f975806df90 2024-03-29T10:59:25.958 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.978+0000 7f976affd640 1 -- 172.21.15.112:0/1341221446 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9764050340 con 0x7f97740a94e0 2024-03-29T10:59:25.958 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.978+0000 7f976affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:25.958 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:25.978+0000 7f976affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:26.292 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:26.310+0000 7f977c704640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:26.292 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:26.310+0000 7f977c704640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:26.292 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:26.310+0000 7f977c704640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:59:26.292 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:26.310+0000 7f977c704640 1 -- 172.21.15.112:0/1341221446 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f975c005330 con 0x7f97740a94e0 2024-03-29T10:59:26.294 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:26.314+0000 7f976affd640 1 -- 172.21.15.112:0/1341221446 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f9764050340 con 0x7f97740a94e0 2024-03-29T10:59:26.294 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:26.314+0000 7f976affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:26.294 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:26.314+0000 7f976affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:26.294 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:59:26.295 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:26.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:26.317+0000 7f977c704640 10 librados: watch_flush enter 2024-03-29T10:59:26.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:26.317+0000 7f977c704640 10 librados: watch_flush exit 2024-03-29T10:59:26.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:26.318+0000 7f977c704640 20 client.4550.objecter shutdown clearing up homeless session... 2024-03-29T10:59:26.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:26.318+0000 7f977c704640 10 client.4550.objecter successfully canceled tick 2024-03-29T10:59:26.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:26.318+0000 7f977c704640 1 -- 172.21.15.112:0/1341221446 >> v1:172.21.15.118:6816/3181073408 conn(0x7f975806df90 legacy=0x7f9758070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:26.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:26.318+0000 7f977c704640 10 monclient: shutdown 2024-03-29T10:59:26.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:26.318+0000 7f977c704640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:26.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:26.318+0000 7f977c704640 1 -- 172.21.15.112:0/1341221446 >> v1:172.21.15.112:6790/0 conn(0x7f97740a94e0 legacy=0x7f977413e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:26.300 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:26.319+0000 7f977c704640 1 -- 172.21.15.112:0/1341221446 shutdown_connections 2024-03-29T10:59:26.300 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:26.319+0000 7f977c704640 1 -- 172.21.15.112:0/1341221446 >> 172.21.15.112:0/1341221446 conn(0x7f97740b3a00 msgr2=0x7f977413ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:26.300 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:26.319+0000 7f977c704640 1 -- 172.21.15.112:0/1341221446 shutdown_connections 2024-03-29T10:59:26.300 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:26.319+0000 7f977c704640 1 -- 172.21.15.112:0/1341221446 wait complete. 2024-03-29T10:59:26.300 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:26.319+0000 7f977c704640 1 librados: shutdown 2024-03-29T10:59:26.319 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:33.124 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:33.291 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.310+0000 7fa9d6671640 10 monclient: get_monmap_and_config 2024-03-29T10:59:33.291 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.310+0000 7fa9d6671640 10 monclient: build_initial_monmap 2024-03-29T10:59:33.291 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.310+0000 7fa9d6671640 10 monclient: monmap: 2024-03-29T10:59:33.291 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:59:33.291 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:33.291 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:59:33.311765+0000 2024-03-29T10:59:33.291 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:59:33.311765+0000 2024-03-29T10:59:33.291 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:33.291 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:33.291 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:59:33.292 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:59:33.292 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:59:33.292 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:33.292 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.311+0000 7fa9d6671640 1 Processor -- start 2024-03-29T10:59:33.292 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.311+0000 7fa9d6671640 1 -- start start 2024-03-29T10:59:33.292 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.311+0000 7fa9d6671640 10 monclient: init 2024-03-29T10:59:33.293 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.313+0000 7fa9d6671640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:33.293 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.313+0000 7fa9d6671640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:59:33.293 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.313+0000 7fa9d566f640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fa9d00b2800 0x7fa9d0142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:54150/0 (socket says 172.21.15.112:54150) 2024-03-29T10:59:33.294 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.313+0000 7fa9d566f640 1 -- 172.21.15.112:0/2586101858 learned_addr learned my addr 172.21.15.112:0/2586101858 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:59:33.294 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.313+0000 7fa9d6671640 10 monclient(hunting): picked mon.noname-b con 0x7fa9d00b2800 addr v1:172.21.15.112:6790/0 2024-03-29T10:59:33.294 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.313+0000 7fa9d6671640 10 monclient(hunting): picked mon.noname-c con 0x7fa9d00a9c00 addr v1:172.21.15.118:6789/0 2024-03-29T10:59:33.297 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.317+0000 7fa9d6671640 10 monclient(hunting): picked mon.noname-a con 0x7fa9d00b38c0 addr v1:172.21.15.112:6789/0 2024-03-29T10:59:33.297 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.317+0000 7fa9d6671640 1 -- 172.21.15.112:0/2586101858 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d0141eb0 con 0x7fa9d00b38c0 2024-03-29T10:59:33.297 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.317+0000 7fa9d6671640 1 -- 172.21.15.112:0/2586101858 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d00b22d0 con 0x7fa9d00a9c00 2024-03-29T10:59:33.297 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.317+0000 7fa9d6671640 1 -- 172.21.15.112:0/2586101858 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d0142d30 con 0x7fa9d00b2800 2024-03-29T10:59:33.297 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.317+0000 7fa9d6671640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:33.297 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.317+0000 7fa9d6671640 10 monclient(hunting): authenticate will time out at 1328.621426s 2024-03-29T10:59:33.297 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.317+0000 7fa9c3fff640 1 -- 172.21.15.112:0/2586101858 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091733020 0 0) 0x7fa9d00b22d0 con 0x7fa9d00a9c00 2024-03-29T10:59:33.298 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.317+0000 7fa9c3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:33.298 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.317+0000 7fa9c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:33.298 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.317+0000 7fa9c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:33.298 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.317+0000 7fa9c3fff640 10 monclient(hunting): my global_id is 4492 2024-03-29T10:59:33.298 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.317+0000 7fa9c3fff640 1 -- 172.21.15.112:0/2586101858 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9bc003520 con 0x7fa9d00a9c00 2024-03-29T10:59:33.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.318+0000 7fa9c3fff640 1 -- 172.21.15.112:0/2586101858 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1110387863 0 0) 0x7fa9bc003520 con 0x7fa9d00a9c00 2024-03-29T10:59:33.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.318+0000 7fa9c3fff640 1 -- 172.21.15.112:0/2586101858 >> v1:172.21.15.112:6790/0 conn(0x7fa9d00b2800 legacy=0x7fa9d0142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:33.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.318+0000 7fa9c3fff640 1 -- 172.21.15.112:0/2586101858 >> v1:172.21.15.112:6789/0 conn(0x7fa9d00b38c0 legacy=0x7fa9d014d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:59:33.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.318+0000 7fa9c3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:59:33.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.318+0000 7fa9c3fff640 1 monclient: found mon.noname-c 2024-03-29T10:59:33.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.318+0000 7fa9c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:33.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.318+0000 7fa9c3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T10:59:33.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.318+0000 7fa9c3fff640 1 -- 172.21.15.112:0/2586101858 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d00aaf90 con 0x7fa9d00a9c00 2024-03-29T10:59:33.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.318+0000 7fa9c3fff640 10 monclient: _finish_auth 0 2024-03-29T10:59:33.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.318+0000 7fa9c3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:59:33.299 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.318+0000 7fa9c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:33.300 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.318+0000 7fa9c3fff640 1 -- 172.21.15.112:0/2586101858 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fa9cc001690 con 0x7fa9d00a9c00 2024-03-29T10:59:33.300 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.318+0000 7fa9c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:33.300 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.318+0000 7fa9c3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:33.300 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.318+0000 7fa9c3fff640 10 monclient: dump: 2024-03-29T10:59:33.300 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:33.300 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:33.300 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:33.300 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:33.300 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:33.300 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:33.300 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:33.300 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:33.300 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:33.300 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:33.301 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.318+0000 7fa9d6671640 5 monclient: authenticate success, global_id 4492 2024-03-29T10:59:33.301 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.318+0000 7fa9d6671640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:33.301 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.318+0000 7fa9c3fff640 1 -- 172.21.15.112:0/2586101858 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9cc0018b0 con 0x7fa9d00a9c00 2024-03-29T10:59:33.301 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.318+0000 7fa9c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:33.301 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.318+0000 7fa9d6671640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:33.301 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.318+0000 7fa9d6671640 4 set_mon_vals no callback set 2024-03-29T10:59:33.301 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.319+0000 7fa9c3fff640 1 -- 172.21.15.112:0/2586101858 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fa9cc001c40 con 0x7fa9d00a9c00 2024-03-29T10:59:33.301 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.319+0000 7fa9c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:33.301 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.319+0000 7fa9c3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:33.301 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.319+0000 7fa9c3fff640 10 monclient: dump: 2024-03-29T10:59:33.301 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:33.301 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:33.301 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:33.301 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:33.301 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:33.302 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:33.302 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:33.302 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:33.302 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:33.302 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:33.304 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.319+0000 7fa9d6671640 10 monclient: shutdown 2024-03-29T10:59:33.304 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.319+0000 7fa9d6671640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:33.304 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.319+0000 7fa9d6671640 1 -- 172.21.15.112:0/2586101858 >> v1:172.21.15.118:6789/0 conn(0x7fa9d00a9c00 legacy=0x7fa9d013bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:33.304 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.319+0000 7fa9d6671640 1 -- 172.21.15.112:0/2586101858 shutdown_connections 2024-03-29T10:59:33.304 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.319+0000 7fa9d6671640 1 -- 172.21.15.112:0/2586101858 >> 172.21.15.112:0/2586101858 conn(0x7fa9d0066a00 msgr2=0x7fa9d0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:33.304 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.320+0000 7fa9d6671640 1 -- 172.21.15.112:0/2586101858 shutdown_connections 2024-03-29T10:59:33.304 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.320+0000 7fa9d6671640 1 -- 172.21.15.112:0/2586101858 wait complete. 2024-03-29T10:59:33.304 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.321+0000 7fa9d6671640 10 monclient: build_initial_monmap 2024-03-29T10:59:33.304 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.321+0000 7fa9d6671640 10 monclient: monmap: 2024-03-29T10:59:33.305 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:59:33.305 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:33.305 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:59:33.305 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:59:33.305 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:33.305 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:33.305 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:59:33.305 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:59:33.305 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:59:33.305 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:33.305 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.322+0000 7fa9d6671640 1 librados: starting msgr at 2024-03-29T10:59:33.305 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.322+0000 7fa9d6671640 1 librados: starting objecter 2024-03-29T10:59:33.305 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.322+0000 7fa9d6671640 1 Processor -- start 2024-03-29T10:59:33.305 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.323+0000 7fa9d6671640 1 -- start start 2024-03-29T10:59:33.306 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.323+0000 7fa9d6671640 1 librados: setting wanted keys 2024-03-29T10:59:33.306 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.323+0000 7fa9d6671640 1 librados: calling monclient init 2024-03-29T10:59:33.306 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.323+0000 7fa9d6671640 10 monclient: init 2024-03-29T10:59:33.306 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.324+0000 7fa9d6671640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:33.306 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.324+0000 7fa9d6671640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:59:33.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.327+0000 7fa9d6671640 10 monclient(hunting): picked mon.noname-a con 0x7fa9d00a9c00 addr v1:172.21.15.112:6789/0 2024-03-29T10:59:33.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.327+0000 7fa9d566f640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fa9d00a9c00 0x7fa9d01d72a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:44647/0 (socket says 172.21.15.112:44647) 2024-03-29T10:59:33.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.327+0000 7fa9d566f640 1 -- 172.21.15.112:0/2382916865 learned_addr learned my addr 172.21.15.112:0/2382916865 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:59:33.307 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.327+0000 7fa9d6671640 10 monclient(hunting): picked mon.noname-c con 0x7fa9d00b2800 addr v1:172.21.15.112:6790/0 2024-03-29T10:59:33.310 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.330+0000 7fa9c1ffb640 10 client.?.objecter ms_handle_connect 0x7fa9d00a9c00 2024-03-29T10:59:33.310 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.330+0000 7fa9d6671640 10 monclient(hunting): picked mon.noname-b con 0x7fa9d00b38c0 addr v1:172.21.15.118:6789/0 2024-03-29T10:59:33.310 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.330+0000 7fa9c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:33.310 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.330+0000 7fa9d6671640 1 -- 172.21.15.112:0/2382916865 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d00acc40 con 0x7fa9d00a9c00 2024-03-29T10:59:33.310 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.330+0000 7fa9d6671640 1 -- 172.21.15.112:0/2382916865 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d0138710 con 0x7fa9d00b38c0 2024-03-29T10:59:33.310 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.330+0000 7fa9d6671640 1 -- 172.21.15.112:0/2382916865 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d013c480 con 0x7fa9d00b2800 2024-03-29T10:59:33.311 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.330+0000 7fa9d6671640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:33.311 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.330+0000 7fa9d6671640 10 monclient(hunting): authenticate will time out at 1328.634702s 2024-03-29T10:59:33.311 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.330+0000 7fa9c1ffb640 10 client.?.objecter ms_handle_connect 0x7fa9d00b38c0 2024-03-29T10:59:33.311 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.330+0000 7fa9c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:33.311 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.330+0000 7fa9c1ffb640 10 client.?.objecter ms_handle_connect 0x7fa9d00b2800 2024-03-29T10:59:33.311 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.330+0000 7fa9c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:33.312 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.330+0000 7fa9c1ffb640 1 -- 172.21.15.112:0/2382916865 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3380288226 0 0) 0x7fa9d00acc40 con 0x7fa9d00a9c00 2024-03-29T10:59:33.312 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.330+0000 7fa9c1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:59:33.312 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.330+0000 7fa9c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:33.312 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.330+0000 7fa9c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:33.312 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.331+0000 7fa9c1ffb640 10 monclient(hunting): my global_id is 4518 2024-03-29T10:59:33.312 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.331+0000 7fa9c1ffb640 1 -- 172.21.15.112:0/2382916865 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9b4004eb0 con 0x7fa9d00a9c00 2024-03-29T10:59:33.312 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.331+0000 7fa9c1ffb640 1 -- 172.21.15.112:0/2382916865 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1525982897 0 0) 0x7fa9d0138710 con 0x7fa9d00b38c0 2024-03-29T10:59:33.312 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.331+0000 7fa9c1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:59:33.312 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.331+0000 7fa9c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:33.312 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.331+0000 7fa9c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:33.312 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.331+0000 7fa9c1ffb640 10 monclient(hunting): my global_id is 4495 2024-03-29T10:59:33.312 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.331+0000 7fa9c1ffb640 1 -- 172.21.15.112:0/2382916865 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d00acc40 con 0x7fa9d00b38c0 2024-03-29T10:59:33.312 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.331+0000 7fa9c1ffb640 1 -- 172.21.15.112:0/2382916865 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4228709022 0 0) 0x7fa9d00acc40 con 0x7fa9d00b38c0 2024-03-29T10:59:33.313 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.331+0000 7fa9c1ffb640 1 -- 172.21.15.112:0/2382916865 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9d0138710 con 0x7fa9d00b38c0 2024-03-29T10:59:33.313 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.331+0000 7fa9c1ffb640 1 -- 172.21.15.112:0/2382916865 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fa9c8001a70 con 0x7fa9d00b38c0 2024-03-29T10:59:33.313 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.331+0000 7fa9c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:33.313 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.331+0000 7fa9c1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:33.313 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.331+0000 7fa9c1ffb640 10 monclient(hunting): dump: 2024-03-29T10:59:33.313 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:33.313 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:33.313 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:33.313 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:33.313 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:33.313 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:33.313 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:33.313 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:33.313 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:33.313 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:33.314 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.331+0000 7fa9c1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:33.314 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.332+0000 7fa9c1ffb640 1 -- 172.21.15.112:0/2382916865 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3607313355 0 0) 0x7fa9b4004eb0 con 0x7fa9d00a9c00 2024-03-29T10:59:33.315 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.332+0000 7fa9c1ffb640 1 -- 172.21.15.112:0/2382916865 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9d00acc40 con 0x7fa9d00a9c00 2024-03-29T10:59:33.315 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.332+0000 7fa9c1ffb640 1 -- 172.21.15.112:0/2382916865 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fa9c40018b0 con 0x7fa9d00a9c00 2024-03-29T10:59:33.315 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.332+0000 7fa9c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:33.315 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.332+0000 7fa9c1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:33.315 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.332+0000 7fa9c1ffb640 10 monclient(hunting): dump: 2024-03-29T10:59:33.315 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:33.315 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:33.315 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:33.315 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:33.315 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:33.315 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:33.315 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:33.315 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:33.316 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:33.316 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:33.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.332+0000 7fa9c1ffb640 1 -- 172.21.15.112:0/2382916865 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 701574649 0 0) 0x7fa9d0138710 con 0x7fa9d00b38c0 2024-03-29T10:59:33.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.332+0000 7fa9c1ffb640 1 -- 172.21.15.112:0/2382916865 >> v1:172.21.15.112:6790/0 conn(0x7fa9d00b2800 legacy=0x7fa9d01378f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:33.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.332+0000 7fa9c1ffb640 1 -- 172.21.15.112:0/2382916865 >> v1:172.21.15.112:6789/0 conn(0x7fa9d00a9c00 legacy=0x7fa9d01d72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:33.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.332+0000 7fa9c1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:59:33.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.332+0000 7fa9c1ffb640 1 monclient: found mon.b 2024-03-29T10:59:33.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.332+0000 7fa9c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:33.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.332+0000 7fa9c1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:59:33.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.332+0000 7fa9c1ffb640 1 -- 172.21.15.112:0/2382916865 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d013c610 con 0x7fa9d00b38c0 2024-03-29T10:59:33.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.332+0000 7fa9c1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:59:33.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.332+0000 7fa9c1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:59:33.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.332+0000 7fa9c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:33.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.332+0000 7fa9c1ffb640 1 -- 172.21.15.112:0/2382916865 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9c8001f60 con 0x7fa9d00b38c0 2024-03-29T10:59:33.316 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.332+0000 7fa9c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:33.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.332+0000 7fa9c1ffb640 1 -- 172.21.15.112:0/2382916865 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fa9c8003970 con 0x7fa9d00b38c0 2024-03-29T10:59:33.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.332+0000 7fa9c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:33.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.332+0000 7fa9c1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:33.317 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.332+0000 7fa9c1ffb640 10 monclient: dump: 2024-03-29T10:59:33.317 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:33.317 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:33.317 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:33.317 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:33.317 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:33.317 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:33.317 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:33.317 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:33.317 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:33.317 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:33.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.332+0000 7fa9c37fe640 4 set_mon_vals no callback set 2024-03-29T10:59:33.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.333+0000 7fa9d6671640 5 monclient: authenticate success, global_id 4495 2024-03-29T10:59:33.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.333+0000 7fa9d6671640 10 monclient: _renew_subs 2024-03-29T10:59:33.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.333+0000 7fa9d6671640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:59:33.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.333+0000 7fa9d6671640 1 -- 172.21.15.112:0/2382916865 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d00abbf0 con 0x7fa9d00b38c0 2024-03-29T10:59:33.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.334+0000 7fa9d6671640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:33.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.334+0000 7fa9d6671640 10 monclient: _renew_subs 2024-03-29T10:59:33.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.334+0000 7fa9d6671640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:59:33.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.334+0000 7fa9d6671640 1 -- 172.21.15.112:0/2382916865 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d01d7b90 con 0x7fa9d00b38c0 2024-03-29T10:59:33.318 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.334+0000 7fa9d6671640 1 librados: init done 2024-03-29T10:59:33.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.337+0000 7fa9c1ffb640 1 -- 172.21.15.112:0/2382916865 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fa9c8003ad0 con 0x7fa9d00b38c0 2024-03-29T10:59:33.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.338+0000 7fa9d566f640 10 monclient: get_auth_request con 0x7fa9b406dee0 auth_method 0 2024-03-29T10:59:33.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.338+0000 7fa9c1ffb640 1 -- 172.21.15.112:0/2382916865 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7fa9c8083830 con 0x7fa9d00b38c0 2024-03-29T10:59:33.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.338+0000 7fa9c1ffb640 10 client.4495.objecter ms_dispatch 0x7fa9d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:33.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.338+0000 7fa9c1ffb640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:33.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.338+0000 7fa9c1ffb640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:33.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.338+0000 7fa9c1ffb640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-29T10:59:33.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.338+0000 7fa9c1ffb640 10 client.4495.objecter ms_handle_connect 0x7fa9b406dee0 2024-03-29T10:59:33.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.338+0000 7fa9d6671640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:33.319 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.338+0000 7fa9d6671640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:33.320 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.338+0000 7fa9d6671640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:59:33.320 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.338+0000 7fa9d6671640 1 -- 172.21.15.112:0/2382916865 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9d004f730 con 0x7fa9d00b38c0 2024-03-29T10:59:33.325 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.346+0000 7fa9c1ffb640 1 -- 172.21.15.112:0/2382916865 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa9c804f110 con 0x7fa9d00b38c0 2024-03-29T10:59:33.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.346+0000 7fa9c1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:33.326 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.346+0000 7fa9c1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:33.678 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:59:33.678 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:33.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.697+0000 7fa9d6671640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:33.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.697+0000 7fa9d6671640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:33.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.697+0000 7fa9d6671640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:59:33.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.697+0000 7fa9d6671640 1 -- 172.21.15.112:0/2382916865 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9d01d6ea0 con 0x7fa9d00b38c0 2024-03-29T10:59:33.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.698+0000 7fa9c1ffb640 1 -- 172.21.15.112:0/2382916865 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7fa9c8051040 con 0x7fa9d00b38c0 2024-03-29T10:59:33.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.698+0000 7fa9c1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:33.679 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.698+0000 7fa9c1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:33.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.701+0000 7fa9d6671640 10 librados: watch_flush enter 2024-03-29T10:59:33.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.701+0000 7fa9d6671640 10 librados: watch_flush exit 2024-03-29T10:59:33.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.701+0000 7fa9d6671640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-29T10:59:33.681 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.701+0000 7fa9d6671640 10 client.4495.objecter successfully canceled tick 2024-03-29T10:59:33.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.702+0000 7fa9d6671640 1 -- 172.21.15.112:0/2382916865 >> v1:172.21.15.118:6816/3181073408 conn(0x7fa9b406dee0 legacy=0x7fa9b40703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:33.682 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.702+0000 7fa9d6671640 10 monclient: shutdown 2024-03-29T10:59:33.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.702+0000 7fa9d6671640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:33.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.702+0000 7fa9d6671640 1 -- 172.21.15.112:0/2382916865 >> v1:172.21.15.118:6789/0 conn(0x7fa9d00b38c0 legacy=0x7fa9d0138000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:33.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.702+0000 7fa9d6671640 1 -- 172.21.15.112:0/2382916865 shutdown_connections 2024-03-29T10:59:33.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.702+0000 7fa9d6671640 1 -- 172.21.15.112:0/2382916865 >> 172.21.15.112:0/2382916865 conn(0x7fa9d0066a00 msgr2=0x7fa9d0140570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:33.683 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.703+0000 7fa9d6671640 1 -- 172.21.15.112:0/2382916865 shutdown_connections 2024-03-29T10:59:33.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.703+0000 7fa9d6671640 1 -- 172.21.15.112:0/2382916865 wait complete. 2024-03-29T10:59:33.686 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:33.703+0000 7fa9d6671640 1 librados: shutdown 2024-03-29T10:59:33.699 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:40.506 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:40.660 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.679+0000 7f76af59e640 10 monclient: get_monmap_and_config 2024-03-29T10:59:40.660 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.679+0000 7f76af59e640 10 monclient: build_initial_monmap 2024-03-29T10:59:40.660 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.679+0000 7f76af59e640 10 monclient: monmap: 2024-03-29T10:59:40.660 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:59:40.660 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:40.661 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:59:40.680339+0000 2024-03-29T10:59:40.661 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:59:40.680339+0000 2024-03-29T10:59:40.661 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:40.661 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:40.661 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:59:40.661 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:59:40.661 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:59:40.661 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:40.661 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.680+0000 7f76af59e640 1 Processor -- start 2024-03-29T10:59:40.661 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.680+0000 7f76af59e640 1 -- start start 2024-03-29T10:59:40.661 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.680+0000 7f76af59e640 10 monclient: init 2024-03-29T10:59:40.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.681+0000 7f76af59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:40.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.681+0000 7f76af59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:59:40.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.681+0000 7f76af59e640 10 monclient(hunting): picked mon.noname-a con 0x7f76a802cc50 addr v1:172.21.15.112:6789/0 2024-03-29T10:59:40.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.681+0000 7f76af59e640 10 monclient(hunting): picked mon.noname-c con 0x7f76a802d3a0 addr v1:172.21.15.118:6789/0 2024-03-29T10:59:40.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.681+0000 7f76af59e640 10 monclient(hunting): picked mon.noname-b con 0x7f76a80c6290 addr v1:172.21.15.112:6790/0 2024-03-29T10:59:40.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.681+0000 7f76af59e640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a80c56f0 con 0x7f76a802cc50 2024-03-29T10:59:40.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.681+0000 7f76af59e640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a802e080 con 0x7f76a802d3a0 2024-03-29T10:59:40.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.681+0000 7f76af59e640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a802e360 con 0x7f76a80c6290 2024-03-29T10:59:40.665 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.681+0000 7f76af59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:40.666 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.681+0000 7f76af59e640 10 monclient(hunting): authenticate will time out at 1335.985844s 2024-03-29T10:59:40.666 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.685+0000 7f76ae59c640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f76a802cc50 0x7f76a80c2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46307/0 (socket says 172.21.15.112:46307) 2024-03-29T10:59:40.666 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.685+0000 7f76ae59c640 1 -- 172.21.15.112:0/100685695 learned_addr learned my addr 172.21.15.112:0/100685695 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:59:40.666 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.686+0000 7f76ad59a640 1 -- 172.21.15.112:0/100685695 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348363858 0 0) 0x7f76a80c56f0 con 0x7f76a802cc50 2024-03-29T10:59:40.667 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.686+0000 7f76ad59a640 10 monclient(hunting): _negotiate 2024-03-29T10:59:40.667 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.686+0000 7f76ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:40.668 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.686+0000 7f76ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:40.668 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.686+0000 7f76ad59a640 10 monclient(hunting): my global_id is 4521 2024-03-29T10:59:40.668 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.686+0000 7f76ad59a640 1 -- 172.21.15.112:0/100685695 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7698003520 con 0x7f76a802cc50 2024-03-29T10:59:40.668 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.686+0000 7f76ad59a640 1 -- 172.21.15.112:0/100685695 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437423076 0 0) 0x7f76a802e360 con 0x7f76a80c6290 2024-03-29T10:59:40.668 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.686+0000 7f76ad59a640 10 monclient(hunting): _negotiate 2024-03-29T10:59:40.668 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.686+0000 7f76ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:40.668 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.686+0000 7f76ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:40.668 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.686+0000 7f76ad59a640 10 monclient(hunting): my global_id is 4559 2024-03-29T10:59:40.668 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.686+0000 7f76ad59a640 1 -- 172.21.15.112:0/100685695 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76a80c56f0 con 0x7f76a80c6290 2024-03-29T10:59:40.668 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.687+0000 7f76ad59a640 1 -- 172.21.15.112:0/100685695 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1095825149 0 0) 0x7f76a80c56f0 con 0x7f76a80c6290 2024-03-29T10:59:40.668 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.687+0000 7f76ad59a640 1 -- 172.21.15.112:0/100685695 >> v1:172.21.15.118:6789/0 conn(0x7f76a802d3a0 legacy=0x7f76a80c5b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:59:40.668 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.687+0000 7f76ad59a640 1 -- 172.21.15.112:0/100685695 >> v1:172.21.15.112:6789/0 conn(0x7f76a802cc50 legacy=0x7f76a80c2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:40.668 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.687+0000 7f76ad59a640 10 monclient: _finish_hunting 0 2024-03-29T10:59:40.668 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.687+0000 7f76ad59a640 1 monclient: found mon.noname-b 2024-03-29T10:59:40.668 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.687+0000 7f76ad59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:40.669 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.687+0000 7f76ad59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:59:40.669 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.687+0000 7f76ad59a640 1 -- 172.21.15.112:0/100685695 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a80c8c10 con 0x7f76a80c6290 2024-03-29T10:59:40.669 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.687+0000 7f76ad59a640 10 monclient: _finish_auth 0 2024-03-29T10:59:40.669 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.687+0000 7f76ad59a640 10 monclient: _check_auth_tickets 2024-03-29T10:59:40.669 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.687+0000 7f76ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:40.669 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.687+0000 7f76ad59a640 1 -- 172.21.15.112:0/100685695 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f76b004f730 con 0x7f76a80c6290 2024-03-29T10:59:40.669 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.687+0000 7f76ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:40.669 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.687+0000 7f76ad59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:40.669 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.687+0000 7f76ad59a640 10 monclient: dump: 2024-03-29T10:59:40.669 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:40.669 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:40.669 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:40.669 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:40.669 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:40.669 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:40.670 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:40.670 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:40.670 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:40.670 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:40.670 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.687+0000 7f76af59e640 5 monclient: authenticate success, global_id 4559 2024-03-29T10:59:40.670 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.687+0000 7f76af59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:40.670 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.690+0000 7f76ad59a640 1 -- 172.21.15.112:0/100685695 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76b00008d0 con 0x7f76a80c6290 2024-03-29T10:59:40.670 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.690+0000 7f76ad59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:40.670 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.690+0000 7f76ad59a640 1 -- 172.21.15.112:0/100685695 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f76b00758b0 con 0x7f76a80c6290 2024-03-29T10:59:40.671 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.690+0000 7f76ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:40.671 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.690+0000 7f76ad59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:40.671 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.690+0000 7f76ad59a640 10 monclient: dump: 2024-03-29T10:59:40.671 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:40.671 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:40.671 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:40.671 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:40.671 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:40.671 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:40.671 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:40.671 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:40.671 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:40.671 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:40.672 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.691+0000 7f76af59e640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:40.672 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.691+0000 7f76af59e640 4 set_mon_vals no callback set 2024-03-29T10:59:40.672 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.692+0000 7f76af59e640 10 monclient: shutdown 2024-03-29T10:59:40.672 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.692+0000 7f76af59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:40.672 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.692+0000 7f76af59e640 1 -- 172.21.15.112:0/100685695 >> v1:172.21.15.112:6790/0 conn(0x7f76a80c6290 legacy=0x7f76a80d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:40.672 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.692+0000 7f76af59e640 1 -- 172.21.15.112:0/100685695 shutdown_connections 2024-03-29T10:59:40.673 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.692+0000 7f76af59e640 1 -- 172.21.15.112:0/100685695 >> 172.21.15.112:0/100685695 conn(0x7f76a80058f0 msgr2=0x7f76a8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:40.673 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.692+0000 7f76af59e640 1 -- 172.21.15.112:0/100685695 shutdown_connections 2024-03-29T10:59:40.673 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.693+0000 7f76af59e640 1 -- 172.21.15.112:0/100685695 wait complete. 2024-03-29T10:59:40.674 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.694+0000 7f76af59e640 10 monclient: build_initial_monmap 2024-03-29T10:59:40.675 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.694+0000 7f76af59e640 10 monclient: monmap: 2024-03-29T10:59:40.675 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:59:40.675 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:40.675 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:59:40.675 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:59:40.675 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:40.675 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:40.675 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:59:40.675 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:59:40.675 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:59:40.675 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:40.675 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.694+0000 7f76af59e640 1 librados: starting msgr at 2024-03-29T10:59:40.675 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.694+0000 7f76af59e640 1 librados: starting objecter 2024-03-29T10:59:40.675 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.694+0000 7f76af59e640 1 Processor -- start 2024-03-29T10:59:40.675 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.695+0000 7f76af59e640 1 -- start start 2024-03-29T10:59:40.676 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.695+0000 7f76af59e640 1 librados: setting wanted keys 2024-03-29T10:59:40.676 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.695+0000 7f76af59e640 1 librados: calling monclient init 2024-03-29T10:59:40.676 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.695+0000 7f76af59e640 10 monclient: init 2024-03-29T10:59:40.676 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.695+0000 7f76af59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:40.676 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.695+0000 7f76af59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:59:40.676 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.695+0000 7f76af59e640 10 monclient(hunting): picked mon.noname-b con 0x7f76a802cc50 addr v1:172.21.15.118:6789/0 2024-03-29T10:59:40.676 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.695+0000 7f76af59e640 10 monclient(hunting): picked mon.noname-a con 0x7f76a802d3a0 addr v1:172.21.15.112:6789/0 2024-03-29T10:59:40.676 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.695+0000 7f76af59e640 10 monclient(hunting): picked mon.noname-c con 0x7f76a80c6290 addr v1:172.21.15.112:6790/0 2024-03-29T10:59:40.676 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.695+0000 7f76af59e640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a8032a60 con 0x7f76a802d3a0 2024-03-29T10:59:40.676 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.695+0000 7f76af59e640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a80327c0 con 0x7f76a802cc50 2024-03-29T10:59:40.676 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.695+0000 7f76af59e640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a8031de0 con 0x7f76a80c6290 2024-03-29T10:59:40.677 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.695+0000 7f76af59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:40.677 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.695+0000 7f76af59e640 10 monclient(hunting): authenticate will time out at 1336.000096s 2024-03-29T10:59:40.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.700+0000 7f76add9b640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f76a802d3a0 0x7f76a8030870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:48423/0 (socket says 172.21.15.112:48423) 2024-03-29T10:59:40.680 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.700+0000 7f76add9b640 1 -- 172.21.15.112:0/1087785906 learned_addr learned my addr 172.21.15.112:0/1087785906 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:59:40.684 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.703+0000 7f769effd640 10 client.?.objecter ms_handle_connect 0x7f76a802d3a0 2024-03-29T10:59:40.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.703+0000 7f769effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:40.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.703+0000 7f769effd640 10 client.?.objecter ms_handle_connect 0x7f76a802cc50 2024-03-29T10:59:40.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.704+0000 7f769effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:40.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.704+0000 7f769effd640 1 -- 172.21.15.112:0/1087785906 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201011173 0 0) 0x7f76a8032a60 con 0x7f76a802d3a0 2024-03-29T10:59:40.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.704+0000 7f769effd640 10 monclient(hunting): _negotiate 2024-03-29T10:59:40.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.704+0000 7f769effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:40.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.704+0000 7f769effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:40.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.704+0000 7f769effd640 10 monclient(hunting): my global_id is 4524 2024-03-29T10:59:40.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.704+0000 7f769effd640 1 -- 172.21.15.112:0/1087785906 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7690004eb0 con 0x7f76a802d3a0 2024-03-29T10:59:40.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.704+0000 7f769effd640 1 -- 172.21.15.112:0/1087785906 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4002348140 0 0) 0x7f76a80327c0 con 0x7f76a802cc50 2024-03-29T10:59:40.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.704+0000 7f769effd640 10 monclient(hunting): _negotiate 2024-03-29T10:59:40.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.704+0000 7f769effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:40.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.704+0000 7f769effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:40.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.704+0000 7f769effd640 10 monclient(hunting): my global_id is 4498 2024-03-29T10:59:40.685 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.704+0000 7f769effd640 1 -- 172.21.15.112:0/1087785906 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76a8032a60 con 0x7f76a802cc50 2024-03-29T10:59:40.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.705+0000 7f769effd640 1 -- 172.21.15.112:0/1087785906 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1266785360 0 0) 0x7f76a8032a60 con 0x7f76a802cc50 2024-03-29T10:59:40.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.705+0000 7f769effd640 1 -- 172.21.15.112:0/1087785906 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76a80327c0 con 0x7f76a802cc50 2024-03-29T10:59:40.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.705+0000 7f769effd640 1 -- 172.21.15.112:0/1087785906 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f76a00018d0 con 0x7f76a802cc50 2024-03-29T10:59:40.689 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.705+0000 7f769effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:40.690 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.705+0000 7f769effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:40.690 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.705+0000 7f769effd640 10 monclient(hunting): dump: 2024-03-29T10:59:40.690 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:40.690 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:40.690 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:40.690 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:40.690 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:40.690 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:40.690 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:40.690 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:40.690 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:40.690 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:40.690 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.705+0000 7f769effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:40.690 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.705+0000 7f769effd640 1 -- 172.21.15.112:0/1087785906 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3686554544 0 0) 0x7f7690004eb0 con 0x7f76a802d3a0 2024-03-29T10:59:40.690 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.705+0000 7f769effd640 1 -- 172.21.15.112:0/1087785906 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76a8032a60 con 0x7f76a802d3a0 2024-03-29T10:59:40.691 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.705+0000 7f769effd640 1 -- 172.21.15.112:0/1087785906 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f76a40017f0 con 0x7f76a802d3a0 2024-03-29T10:59:40.691 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.705+0000 7f769effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:40.691 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.705+0000 7f769effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:40.691 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.705+0000 7f769effd640 10 monclient(hunting): dump: 2024-03-29T10:59:40.691 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:40.691 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:40.691 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:40.691 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:40.691 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:40.691 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:40.691 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:40.691 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:40.691 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:40.691 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:40.691 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.706+0000 7f769effd640 1 -- 172.21.15.112:0/1087785906 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3712608096 0 0) 0x7f76a80327c0 con 0x7f76a802cc50 2024-03-29T10:59:40.691 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.706+0000 7f769effd640 1 -- 172.21.15.112:0/1087785906 >> v1:172.21.15.112:6790/0 conn(0x7f76a80c6290 legacy=0x7f76a816b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:40.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.706+0000 7f769effd640 1 -- 172.21.15.112:0/1087785906 >> v1:172.21.15.112:6789/0 conn(0x7f76a802d3a0 legacy=0x7f76a8030870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:40.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.706+0000 7f769effd640 10 monclient: _finish_hunting 0 2024-03-29T10:59:40.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.706+0000 7f769effd640 1 monclient: found mon.b 2024-03-29T10:59:40.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.706+0000 7f769effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:40.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.706+0000 7f769effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:59:40.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.706+0000 7f769effd640 1 -- 172.21.15.112:0/1087785906 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a80320c0 con 0x7f76a802cc50 2024-03-29T10:59:40.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.706+0000 7f769effd640 10 monclient: _finish_auth 0 2024-03-29T10:59:40.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.706+0000 7f769effd640 10 monclient: _check_auth_tickets 2024-03-29T10:59:40.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.706+0000 7f769effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:40.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.706+0000 7f76af59e640 5 monclient: authenticate success, global_id 4498 2024-03-29T10:59:40.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.706+0000 7f76af59e640 10 monclient: _renew_subs 2024-03-29T10:59:40.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.706+0000 7f76af59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:59:40.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.706+0000 7f76af59e640 1 -- 172.21.15.112:0/1087785906 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a80323a0 con 0x7f76a802cc50 2024-03-29T10:59:40.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.706+0000 7f76af59e640 10 client.4498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:40.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.706+0000 7f76af59e640 10 monclient: _renew_subs 2024-03-29T10:59:40.692 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.706+0000 7f76af59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:59:40.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.706+0000 7f76af59e640 1 -- 172.21.15.112:0/1087785906 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a8030f80 con 0x7f76a802cc50 2024-03-29T10:59:40.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.706+0000 7f76af59e640 1 librados: init done 2024-03-29T10:59:40.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.707+0000 7f769effd640 1 -- 172.21.15.112:0/1087785906 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76a00047e0 con 0x7f76a802cc50 2024-03-29T10:59:40.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.707+0000 7f769effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:40.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.707+0000 7f76acd99640 4 set_mon_vals no callback set 2024-03-29T10:59:40.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.707+0000 7f76af59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:40.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.708+0000 7f769effd640 1 -- 172.21.15.112:0/1087785906 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f76a0004bf0 con 0x7f76a802cc50 2024-03-29T10:59:40.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.708+0000 7f769effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:40.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.708+0000 7f769effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:40.693 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.708+0000 7f769effd640 10 monclient: dump: 2024-03-29T10:59:40.693 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:40.693 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:40.693 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:40.693 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:40.693 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:40.693 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:40.694 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:40.694 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:40.694 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:40.694 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:40.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.708+0000 7f76af59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:40.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.708+0000 7f76af59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:59:40.694 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.708+0000 7f76af59e640 1 -- 172.21.15.112:0/1087785906 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7694005080 con 0x7f76a802cc50 2024-03-29T10:59:40.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.709+0000 7f769effd640 1 -- 172.21.15.112:0/1087785906 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f76a0004d50 con 0x7f76a802cc50 2024-03-29T10:59:40.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.710+0000 7f769effd640 1 -- 172.21.15.112:0/1087785906 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f76a00848c0 con 0x7f76a802cc50 2024-03-29T10:59:40.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.710+0000 7f769effd640 10 client.4498.objecter ms_dispatch 0x7f76b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:40.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.710+0000 7f769effd640 3 client.4498.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:40.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.710+0000 7f769effd640 3 client.4498.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:40.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.710+0000 7f769effd640 20 client.4498.objecter dump_active .. 0 homeless 2024-03-29T10:59:40.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.710+0000 7f76add9b640 10 monclient: get_auth_request con 0x7f769006df70 auth_method 0 2024-03-29T10:59:40.695 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.711+0000 7f769effd640 10 client.4498.objecter ms_handle_connect 0x7f769006df70 2024-03-29T10:59:40.696 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.716+0000 7f769effd640 1 -- 172.21.15.112:0/1087785906 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76a00501a0 con 0x7f76a802cc50 2024-03-29T10:59:40.696 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.716+0000 7f769effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:40.696 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:40.716+0000 7f769effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:41.069 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:41.088+0000 7f76af59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:41.069 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:41.088+0000 7f76af59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:41.069 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:41.088+0000 7f76af59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T10:59:41.069 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:41.088+0000 7f76af59e640 1 -- 172.21.15.112:0/1087785906 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7694005330 con 0x7f76a802cc50 2024-03-29T10:59:41.069 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:41.090+0000 7f769effd640 1 -- 172.21.15.112:0/1087785906 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f76a00520d0 con 0x7f76a802cc50 2024-03-29T10:59:41.070 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:41.090+0000 7f769effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:41.070 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:41.090+0000 7f769effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:41.070 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:59:41.070 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:41.074 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:41.093+0000 7f76af59e640 10 librados: watch_flush enter 2024-03-29T10:59:41.074 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:41.094+0000 7f76af59e640 10 librados: watch_flush exit 2024-03-29T10:59:41.074 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:41.094+0000 7f76af59e640 20 client.4498.objecter shutdown clearing up homeless session... 2024-03-29T10:59:41.074 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:41.094+0000 7f76af59e640 10 client.4498.objecter successfully canceled tick 2024-03-29T10:59:41.074 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:41.094+0000 7f76af59e640 1 -- 172.21.15.112:0/1087785906 >> v1:172.21.15.118:6816/3181073408 conn(0x7f769006df70 legacy=0x7f7690070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:41.074 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:41.094+0000 7f76af59e640 10 monclient: shutdown 2024-03-29T10:59:41.074 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:41.094+0000 7f76af59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:41.074 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:41.094+0000 7f76af59e640 1 -- 172.21.15.112:0/1087785906 >> v1:172.21.15.118:6789/0 conn(0x7f76a802cc50 legacy=0x7f76a80c4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:41.075 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:41.094+0000 7f76af59e640 1 -- 172.21.15.112:0/1087785906 shutdown_connections 2024-03-29T10:59:41.075 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:41.094+0000 7f76af59e640 1 -- 172.21.15.112:0/1087785906 >> 172.21.15.112:0/1087785906 conn(0x7f76a80058f0 msgr2=0x7f76a8005cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:41.075 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:41.094+0000 7f76af59e640 1 -- 172.21.15.112:0/1087785906 shutdown_connections 2024-03-29T10:59:41.075 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:41.094+0000 7f76af59e640 1 -- 172.21.15.112:0/1087785906 wait complete. 2024-03-29T10:59:41.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:41.098+0000 7f76af59e640 1 librados: shutdown 2024-03-29T10:59:41.096 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:47.903 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:48.057 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.077+0000 7f97e2a55640 10 monclient: get_monmap_and_config 2024-03-29T10:59:48.057 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.077+0000 7f97e2a55640 10 monclient: build_initial_monmap 2024-03-29T10:59:48.058 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.077+0000 7f97e2a55640 10 monclient: monmap: 2024-03-29T10:59:48.058 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:59:48.058 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:48.058 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:59:48.078226+0000 2024-03-29T10:59:48.058 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:59:48.078226+0000 2024-03-29T10:59:48.058 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:48.058 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:48.058 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:59:48.058 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:59:48.058 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:59:48.058 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:48.059 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.078+0000 7f97e2a55640 1 Processor -- start 2024-03-29T10:59:48.059 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.078+0000 7f97e2a55640 1 -- start start 2024-03-29T10:59:48.059 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.078+0000 7f97e2a55640 10 monclient: init 2024-03-29T10:59:48.059 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.079+0000 7f97e2a55640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:48.059 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.079+0000 7f97e2a55640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:59:48.060 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.079+0000 7f97e2a55640 10 monclient(hunting): picked mon.noname-a con 0x7f97dc0b0500 addr v1:172.21.15.112:6789/0 2024-03-29T10:59:48.060 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.079+0000 7f97e2a55640 10 monclient(hunting): picked mon.noname-b con 0x7f97dc0b08e0 addr v1:172.21.15.112:6790/0 2024-03-29T10:59:48.060 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.079+0000 7f97db7fe640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f97dc0b08e0 0x7f97dc13d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:47328/0 (socket says 172.21.15.112:47328) 2024-03-29T10:59:48.062 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.081+0000 7f97e2a55640 10 monclient(hunting): picked mon.noname-c con 0x7f97dc0aeca0 addr v1:172.21.15.118:6789/0 2024-03-29T10:59:48.062 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.082+0000 7f97e2a55640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97dc140620 con 0x7f97dc0b0500 2024-03-29T10:59:48.062 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.082+0000 7f97e2a55640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97dc13daf0 con 0x7f97dc0aeca0 2024-03-29T10:59:48.062 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.082+0000 7f97e2a55640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97dc0ae180 con 0x7f97dc0b08e0 2024-03-29T10:59:48.062 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.082+0000 7f97e2a55640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:48.062 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.082+0000 7f97e2a55640 10 monclient(hunting): authenticate will time out at 1343.386293s 2024-03-29T10:59:48.063 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.082+0000 7f97db7fe640 1 -- 172.21.15.112:0/129619078 learned_addr learned my addr 172.21.15.112:0/129619078 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:59:48.064 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 1 -- 172.21.15.112:0/129619078 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173216866 0 0) 0x7f97dc0ae180 con 0x7f97dc0b08e0 2024-03-29T10:59:48.064 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 10 monclient(hunting): _negotiate 2024-03-29T10:59:48.064 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:48.064 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:48.064 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 10 monclient(hunting): my global_id is 4562 2024-03-29T10:59:48.064 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 1 -- 172.21.15.112:0/129619078 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97c4003520 con 0x7f97dc0b08e0 2024-03-29T10:59:48.064 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 1 -- 172.21.15.112:0/129619078 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 324579359 0 0) 0x7f97c4003520 con 0x7f97dc0b08e0 2024-03-29T10:59:48.064 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 1 -- 172.21.15.112:0/129619078 >> v1:172.21.15.118:6789/0 conn(0x7f97dc0aeca0 legacy=0x7f97dc14b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:48.064 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 1 -- 172.21.15.112:0/129619078 >> v1:172.21.15.112:6789/0 conn(0x7f97dc0b0500 legacy=0x7f97dc13ff10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:59:48.064 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 10 monclient: _finish_hunting 0 2024-03-29T10:59:48.064 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 1 monclient: found mon.noname-b 2024-03-29T10:59:48.064 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:48.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T10:59:48.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 1 -- 172.21.15.112:0/129619078 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97dc0accf0 con 0x7f97dc0b08e0 2024-03-29T10:59:48.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 10 monclient: _finish_auth 0 2024-03-29T10:59:48.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 10 monclient: _check_auth_tickets 2024-03-29T10:59:48.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:48.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 1 -- 172.21.15.112:0/129619078 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f97d0001670 con 0x7f97dc0b08e0 2024-03-29T10:59:48.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:48.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:48.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 10 monclient: dump: 2024-03-29T10:59:48.065 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:48.065 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:48.065 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:48.065 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:48.065 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:48.065 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:48.066 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:48.066 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:48.066 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:48.066 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:48.066 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97e2a55640 5 monclient: authenticate success, global_id 4562 2024-03-29T10:59:48.066 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97e2a55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:48.066 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 1 -- 172.21.15.112:0/129619078 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97d0001900 con 0x7f97dc0b08e0 2024-03-29T10:59:48.066 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:48.066 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97e2a55640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:48.066 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 1 -- 172.21.15.112:0/129619078 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f97d0001cd0 con 0x7f97dc0b08e0 2024-03-29T10:59:48.066 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:48.066 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:48.066 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97daffd640 10 monclient: dump: 2024-03-29T10:59:48.066 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:48.067 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:48.067 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:48.067 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:48.067 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:48.067 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:48.067 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:48.067 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:48.067 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:48.067 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:48.067 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.083+0000 7f97e2a55640 4 set_mon_vals no callback set 2024-03-29T10:59:48.068 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.085+0000 7f97e2a55640 10 monclient: shutdown 2024-03-29T10:59:48.068 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.085+0000 7f97e2a55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:48.069 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.085+0000 7f97e2a55640 1 -- 172.21.15.112:0/129619078 >> v1:172.21.15.112:6790/0 conn(0x7f97dc0b08e0 legacy=0x7f97dc13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:48.069 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.085+0000 7f97e2a55640 1 -- 172.21.15.112:0/129619078 shutdown_connections 2024-03-29T10:59:48.069 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.085+0000 7f97e2a55640 1 -- 172.21.15.112:0/129619078 >> 172.21.15.112:0/129619078 conn(0x7f97dc06b660 msgr2=0x7f97dc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:48.069 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.086+0000 7f97e2a55640 1 -- 172.21.15.112:0/129619078 shutdown_connections 2024-03-29T10:59:48.069 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.087+0000 7f97e2a55640 1 -- 172.21.15.112:0/129619078 wait complete. 2024-03-29T10:59:48.069 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.088+0000 7f97e2a55640 10 monclient: build_initial_monmap 2024-03-29T10:59:48.069 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.088+0000 7f97e2a55640 10 monclient: monmap: 2024-03-29T10:59:48.069 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:59:48.069 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:48.069 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:59:48.069 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:59:48.069 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:48.069 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:48.069 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:59:48.069 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:59:48.070 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:59:48.070 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:48.070 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.088+0000 7f97e2a55640 1 librados: starting msgr at 2024-03-29T10:59:48.070 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.088+0000 7f97e2a55640 1 librados: starting objecter 2024-03-29T10:59:48.070 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.089+0000 7f97e2a55640 1 Processor -- start 2024-03-29T10:59:48.071 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.089+0000 7f97e2a55640 1 -- start start 2024-03-29T10:59:48.071 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.089+0000 7f97e2a55640 1 librados: setting wanted keys 2024-03-29T10:59:48.071 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.089+0000 7f97e2a55640 1 librados: calling monclient init 2024-03-29T10:59:48.071 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.089+0000 7f97e2a55640 10 monclient: init 2024-03-29T10:59:48.071 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.090+0000 7f97e2a55640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:48.071 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.090+0000 7f97e2a55640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:59:48.071 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.090+0000 7f97e2a55640 10 monclient(hunting): picked mon.noname-b con 0x7f97dc0aeca0 addr v1:172.21.15.118:6789/0 2024-03-29T10:59:48.076 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.090+0000 7f97e2a55640 10 monclient(hunting): picked mon.noname-a con 0x7f97dc0b0500 addr v1:172.21.15.112:6789/0 2024-03-29T10:59:48.076 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.093+0000 7f97e2a55640 10 monclient(hunting): picked mon.noname-c con 0x7f97dc070080 addr v1:172.21.15.112:6790/0 2024-03-29T10:59:48.076 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.093+0000 7f97e2a55640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97dc1d72a0 con 0x7f97dc0b0500 2024-03-29T10:59:48.076 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.093+0000 7f97e2a55640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97dc11ed10 con 0x7f97dc0aeca0 2024-03-29T10:59:48.076 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.093+0000 7f97e2a55640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97dc11c920 con 0x7f97dc070080 2024-03-29T10:59:48.076 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.093+0000 7f97e2a55640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:48.076 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.093+0000 7f97e2a55640 10 monclient(hunting): authenticate will time out at 1343.397471s 2024-03-29T10:59:48.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.093+0000 7f97dbfff640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f97dc0aeca0 0x7f97dc1d6980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:47887/0 (socket says 172.21.15.112:47887) 2024-03-29T10:59:48.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.093+0000 7f97dbfff640 1 -- 172.21.15.112:0/846892926 learned_addr learned my addr 172.21.15.112:0/846892926 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:59:48.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.093+0000 7f97d8ff9640 10 client.?.objecter ms_handle_connect 0x7f97dc0aeca0 2024-03-29T10:59:48.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.093+0000 7f97d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:48.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.093+0000 7f97d8ff9640 10 client.?.objecter ms_handle_connect 0x7f97dc070080 2024-03-29T10:59:48.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.093+0000 7f97d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:48.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.093+0000 7f97d8ff9640 1 -- 172.21.15.112:0/846892926 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772536484 0 0) 0x7f97dc11c920 con 0x7f97dc070080 2024-03-29T10:59:48.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.093+0000 7f97d8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:59:48.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.093+0000 7f97d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:48.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.093+0000 7f97d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:48.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.093+0000 7f97d8ff9640 10 monclient(hunting): my global_id is 4565 2024-03-29T10:59:48.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.093+0000 7f97d8ff9640 1 -- 172.21.15.112:0/846892926 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97c8004eb0 con 0x7f97dc070080 2024-03-29T10:59:48.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.093+0000 7f97d8ff9640 1 -- 172.21.15.112:0/846892926 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278545748 0 0) 0x7f97dc11ed10 con 0x7f97dc0aeca0 2024-03-29T10:59:48.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.093+0000 7f97d8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:59:48.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.093+0000 7f97d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:48.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.093+0000 7f97d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:48.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.093+0000 7f97d8ff9640 10 monclient(hunting): my global_id is 4501 2024-03-29T10:59:48.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.093+0000 7f97d8ff9640 1 -- 172.21.15.112:0/846892926 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97dc11c920 con 0x7f97dc0aeca0 2024-03-29T10:59:48.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97d8ff9640 1 -- 172.21.15.112:0/846892926 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3677294426 0 0) 0x7f97c8004eb0 con 0x7f97dc070080 2024-03-29T10:59:48.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97d8ff9640 1 -- 172.21.15.112:0/846892926 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97dc11ed10 con 0x7f97dc070080 2024-03-29T10:59:48.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97d8ff9640 1 -- 172.21.15.112:0/846892926 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f97d4001bf0 con 0x7f97dc070080 2024-03-29T10:59:48.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:48.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97d8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:48.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97d8ff9640 10 monclient(hunting): dump: 2024-03-29T10:59:48.078 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:48.078 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:48.078 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:48.078 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:48.078 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:48.078 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:48.079 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:48.079 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:48.079 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:48.079 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:48.079 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97d8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:48.079 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97d8ff9640 1 -- 172.21.15.112:0/846892926 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 472545896 0 0) 0x7f97dc11c920 con 0x7f97dc0aeca0 2024-03-29T10:59:48.079 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97d8ff9640 1 -- 172.21.15.112:0/846892926 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97c8004eb0 con 0x7f97dc0aeca0 2024-03-29T10:59:48.079 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97d8ff9640 1 -- 172.21.15.112:0/846892926 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f97cc001920 con 0x7f97dc0aeca0 2024-03-29T10:59:48.079 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:48.079 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97d8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:48.079 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97d8ff9640 10 monclient(hunting): dump: 2024-03-29T10:59:48.079 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:48.079 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:48.079 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:48.079 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:48.079 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:48.080 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:48.080 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:48.080 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:48.080 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:48.080 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:48.080 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97d8ff9640 1 -- 172.21.15.112:0/846892926 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3588964507 0 0) 0x7f97dc11ed10 con 0x7f97dc070080 2024-03-29T10:59:48.080 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97d8ff9640 1 -- 172.21.15.112:0/846892926 >> v1:172.21.15.118:6789/0 conn(0x7f97dc0aeca0 legacy=0x7f97dc1d6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:48.080 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97d8ff9640 1 -- 172.21.15.112:0/846892926 >> v1:172.21.15.112:6789/0 conn(0x7f97dc0b0500 legacy=0x7f97dc1d30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:48.080 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97d8ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:59:48.080 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97d8ff9640 1 monclient: found mon.c 2024-03-29T10:59:48.080 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97d8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:48.080 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97d8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:59:48.080 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97d8ff9640 1 -- 172.21.15.112:0/846892926 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97dc1e1af0 con 0x7f97dc070080 2024-03-29T10:59:48.080 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97d8ff9640 10 monclient: _finish_auth 0 2024-03-29T10:59:48.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97d8ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:59:48.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:48.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97e2a55640 5 monclient: authenticate success, global_id 4565 2024-03-29T10:59:48.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97e2a55640 10 monclient: _renew_subs 2024-03-29T10:59:48.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97e2a55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:59:48.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97e2a55640 1 -- 172.21.15.112:0/846892926 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97dc1deac0 con 0x7f97dc070080 2024-03-29T10:59:48.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97e2a55640 10 client.4565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:48.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97e2a55640 10 monclient: _renew_subs 2024-03-29T10:59:48.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97e2a55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:59:48.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97e2a55640 1 -- 172.21.15.112:0/846892926 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f97dc1deff0 con 0x7f97dc070080 2024-03-29T10:59:48.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.094+0000 7f97e2a55640 1 librados: init done 2024-03-29T10:59:48.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.095+0000 7f97d8ff9640 1 -- 172.21.15.112:0/846892926 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97d40020e0 con 0x7f97dc070080 2024-03-29T10:59:48.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.095+0000 7f97d8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:48.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.095+0000 7f97d8ff9640 1 -- 172.21.15.112:0/846892926 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f97d4003b10 con 0x7f97dc070080 2024-03-29T10:59:48.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.095+0000 7f97d8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:48.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.095+0000 7f97d8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:48.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.095+0000 7f97d8ff9640 10 monclient: dump: 2024-03-29T10:59:48.082 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:48.083 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:48.083 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:48.083 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:48.083 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:48.083 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:48.083 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:48.083 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:48.083 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:48.083 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:48.083 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.095+0000 7f97da7fc640 4 set_mon_vals no callback set 2024-03-29T10:59:48.083 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.096+0000 7f97d8ff9640 1 -- 172.21.15.112:0/846892926 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f97d4003d30 con 0x7f97dc070080 2024-03-29T10:59:48.083 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.097+0000 7f97e2a55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:48.083 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.097+0000 7f97e2a55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:48.083 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.097+0000 7f97e2a55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:59:48.083 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.097+0000 7f97e2a55640 1 -- 172.21.15.112:0/846892926 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97bc005080 con 0x7f97dc070080 2024-03-29T10:59:48.083 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.098+0000 7f97dbfff640 10 monclient: get_auth_request con 0x7f97c806de30 auth_method 0 2024-03-29T10:59:48.084 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.099+0000 7f97d8ff9640 1 -- 172.21.15.112:0/846892926 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f97d4083b10 con 0x7f97dc070080 2024-03-29T10:59:48.084 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.099+0000 7f97d8ff9640 10 client.4565.objecter ms_dispatch 0x7f97dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:48.084 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.099+0000 7f97d8ff9640 3 client.4565.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:48.084 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.099+0000 7f97d8ff9640 3 client.4565.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:48.084 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.099+0000 7f97d8ff9640 20 client.4565.objecter dump_active .. 0 homeless 2024-03-29T10:59:48.084 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.099+0000 7f97d8ff9640 10 client.4565.objecter ms_handle_connect 0x7f97c806de30 2024-03-29T10:59:48.086 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.106+0000 7f97d8ff9640 1 -- 172.21.15.112:0/846892926 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f97d404f3f0 con 0x7f97dc070080 2024-03-29T10:59:48.086 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.106+0000 7f97d8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:48.086 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.106+0000 7f97d8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:48.407 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.427+0000 7f97e2a55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:48.407 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.427+0000 7f97e2a55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:48.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.427+0000 7f97e2a55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T10:59:48.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.427+0000 7f97e2a55640 1 -- 172.21.15.112:0/846892926 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97bc005330 con 0x7f97dc070080 2024-03-29T10:59:48.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.431+0000 7f97d8ff9640 1 -- 172.21.15.112:0/846892926 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f97d4051320 con 0x7f97dc070080 2024-03-29T10:59:48.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.431+0000 7f97d8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:48.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.431+0000 7f97d8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:48.412 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:59:48.412 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:48.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.435+0000 7f97e2a55640 10 librados: watch_flush enter 2024-03-29T10:59:48.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.435+0000 7f97e2a55640 10 librados: watch_flush exit 2024-03-29T10:59:48.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.435+0000 7f97e2a55640 20 client.4565.objecter shutdown clearing up homeless session... 2024-03-29T10:59:48.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.435+0000 7f97e2a55640 10 client.4565.objecter successfully canceled tick 2024-03-29T10:59:48.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.435+0000 7f97e2a55640 1 -- 172.21.15.112:0/846892926 >> v1:172.21.15.118:6816/3181073408 conn(0x7f97c806de30 legacy=0x7f97c80702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:48.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.435+0000 7f97e2a55640 10 monclient: shutdown 2024-03-29T10:59:48.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.435+0000 7f97e2a55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:48.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.435+0000 7f97e2a55640 1 -- 172.21.15.112:0/846892926 >> v1:172.21.15.112:6790/0 conn(0x7f97dc070080 legacy=0x7f97dc1d3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:48.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.436+0000 7f97e2a55640 1 -- 172.21.15.112:0/846892926 shutdown_connections 2024-03-29T10:59:48.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.436+0000 7f97e2a55640 1 -- 172.21.15.112:0/846892926 >> 172.21.15.112:0/846892926 conn(0x7f97dc06b660 msgr2=0x7f97dc0b2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:48.417 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.437+0000 7f97e2a55640 1 -- 172.21.15.112:0/846892926 shutdown_connections 2024-03-29T10:59:48.417 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.437+0000 7f97e2a55640 1 -- 172.21.15.112:0/846892926 wait complete. 2024-03-29T10:59:48.417 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:48.437+0000 7f97e2a55640 1 librados: shutdown 2024-03-29T10:59:48.433 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:55.240 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:55.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.406+0000 7fe6c9b15640 10 monclient: get_monmap_and_config 2024-03-29T10:59:55.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.406+0000 7fe6c9b15640 10 monclient: build_initial_monmap 2024-03-29T10:59:55.386 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.406+0000 7fe6c9b15640 10 monclient: monmap: 2024-03-29T10:59:55.386 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:59:55.386 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:55.387 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:59:55.407348+0000 2024-03-29T10:59:55.387 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:59:55.407348+0000 2024-03-29T10:59:55.387 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:55.387 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:55.387 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:59:55.387 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T10:59:55.387 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T10:59:55.387 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:55.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.407+0000 7fe6c9b15640 1 Processor -- start 2024-03-29T10:59:55.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.407+0000 7fe6c9b15640 1 -- start start 2024-03-29T10:59:55.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.407+0000 7fe6c9b15640 10 monclient: init 2024-03-29T10:59:55.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.408+0000 7fe6c9b15640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:55.388 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.408+0000 7fe6c9b15640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:59:55.389 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.408+0000 7fe6c9b15640 10 monclient(hunting): picked mon.noname-c con 0x7fe6c41422b0 addr v1:172.21.15.118:6789/0 2024-03-29T10:59:55.389 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.408+0000 7fe6c9b15640 10 monclient(hunting): picked mon.noname-b con 0x7fe6c4142690 addr v1:172.21.15.112:6790/0 2024-03-29T10:59:55.389 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.408+0000 7fe6c9b15640 10 monclient(hunting): picked mon.noname-a con 0x7fe6c40b17d0 addr v1:172.21.15.112:6789/0 2024-03-29T10:59:55.389 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.408+0000 7fe6c9b15640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c4142c80 con 0x7fe6c40b17d0 2024-03-29T10:59:55.389 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.408+0000 7fe6c9b15640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c40aae40 con 0x7fe6c41422b0 2024-03-29T10:59:55.389 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.408+0000 7fe6c9b15640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c413fbd0 con 0x7fe6c4142690 2024-03-29T10:59:55.389 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.408+0000 7fe6c9b15640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:55.389 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.408+0000 7fe6c9b15640 10 monclient(hunting): authenticate will time out at 1350.712517s 2024-03-29T10:59:55.389 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.408+0000 7fe6c2ffd640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe6c4142690 0x7fe6c40b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:50942/0 (socket says 172.21.15.112:50942) 2024-03-29T10:59:55.389 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.408+0000 7fe6c2ffd640 1 -- 172.21.15.112:0/572825437 learned_addr learned my addr 172.21.15.112:0/572825437 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:59:55.392 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.409+0000 7fe6c27fc640 1 -- 172.21.15.112:0/572825437 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1348308011 0 0) 0x7fe6c40aae40 con 0x7fe6c41422b0 2024-03-29T10:59:55.392 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.409+0000 7fe6c27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:59:55.392 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.409+0000 7fe6c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:55.393 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.409+0000 7fe6c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:55.393 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.409+0000 7fe6c27fc640 10 monclient(hunting): my global_id is 4504 2024-03-29T10:59:55.393 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.409+0000 7fe6c27fc640 1 -- 172.21.15.112:0/572825437 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6ac003520 con 0x7fe6c41422b0 2024-03-29T10:59:55.393 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.409+0000 7fe6c27fc640 1 -- 172.21.15.112:0/572825437 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963174248 0 0) 0x7fe6c413fbd0 con 0x7fe6c4142690 2024-03-29T10:59:55.393 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.409+0000 7fe6c27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:59:55.393 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.409+0000 7fe6c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:55.393 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.409+0000 7fe6c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:55.393 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.409+0000 7fe6c27fc640 10 monclient(hunting): my global_id is 4568 2024-03-29T10:59:55.393 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.409+0000 7fe6c27fc640 1 -- 172.21.15.112:0/572825437 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6c40aae40 con 0x7fe6c4142690 2024-03-29T10:59:55.393 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.409+0000 7fe6c27fc640 1 -- 172.21.15.112:0/572825437 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311277112 0 0) 0x7fe6c4142c80 con 0x7fe6c40b17d0 2024-03-29T10:59:55.393 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.409+0000 7fe6c27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:59:55.393 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.409+0000 7fe6c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:55.393 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.409+0000 7fe6c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:55.393 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.409+0000 7fe6c27fc640 10 monclient(hunting): my global_id is 4527 2024-03-29T10:59:55.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.409+0000 7fe6c27fc640 1 -- 172.21.15.112:0/572825437 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6c413fbd0 con 0x7fe6c40b17d0 2024-03-29T10:59:55.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.409+0000 7fe6c27fc640 1 -- 172.21.15.112:0/572825437 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2321404768 0 0) 0x7fe6ac003520 con 0x7fe6c41422b0 2024-03-29T10:59:55.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.409+0000 7fe6c27fc640 1 -- 172.21.15.112:0/572825437 >> v1:172.21.15.112:6790/0 conn(0x7fe6c4142690 legacy=0x7fe6c40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:55.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.410+0000 7fe6c27fc640 1 -- 172.21.15.112:0/572825437 >> v1:172.21.15.112:6789/0 conn(0x7fe6c40b17d0 legacy=0x7fe6c40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:55.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.410+0000 7fe6c27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:59:55.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.410+0000 7fe6c27fc640 1 monclient: found mon.noname-c 2024-03-29T10:59:55.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.410+0000 7fe6c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:55.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.410+0000 7fe6c27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T10:59:55.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.410+0000 7fe6c27fc640 1 -- 172.21.15.112:0/572825437 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c4151940 con 0x7fe6c41422b0 2024-03-29T10:59:55.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.410+0000 7fe6c27fc640 10 monclient: _finish_auth 0 2024-03-29T10:59:55.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.410+0000 7fe6c27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:59:55.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.410+0000 7fe6c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:55.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.410+0000 7fe6c9b15640 5 monclient: authenticate success, global_id 4504 2024-03-29T10:59:55.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.410+0000 7fe6c27fc640 1 -- 172.21.15.112:0/572825437 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe6b4001690 con 0x7fe6c41422b0 2024-03-29T10:59:55.395 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.410+0000 7fe6c9b15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:55.395 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.410+0000 7fe6c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:55.395 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.410+0000 7fe6c27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:55.395 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.410+0000 7fe6c27fc640 10 monclient: dump: 2024-03-29T10:59:55.395 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:55.395 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:55.395 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:55.395 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:55.395 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:55.395 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:55.395 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:55.395 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:55.395 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:55.395 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:55.395 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.410+0000 7fe6c9b15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:55.395 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.411+0000 7fe6c27fc640 1 -- 172.21.15.112:0/572825437 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6b40018b0 con 0x7fe6c41422b0 2024-03-29T10:59:55.396 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.412+0000 7fe6c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:55.396 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.412+0000 7fe6c27fc640 1 -- 172.21.15.112:0/572825437 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe6b4001bc0 con 0x7fe6c41422b0 2024-03-29T10:59:55.396 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.412+0000 7fe6c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:55.396 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.412+0000 7fe6c27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:55.396 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.412+0000 7fe6c27fc640 10 monclient: dump: 2024-03-29T10:59:55.396 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:55.396 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:55.396 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:55.396 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:55.396 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:55.396 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:55.396 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:55.396 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:55.396 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:55.396 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:55.397 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.412+0000 7fe6c9b15640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:55.397 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.412+0000 7fe6c9b15640 4 set_mon_vals no callback set 2024-03-29T10:59:55.402 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.413+0000 7fe6c9b15640 10 monclient: shutdown 2024-03-29T10:59:55.402 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.413+0000 7fe6c9b15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:55.402 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.413+0000 7fe6c9b15640 1 -- 172.21.15.112:0/572825437 >> v1:172.21.15.118:6789/0 conn(0x7fe6c41422b0 legacy=0x7fe6c413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:55.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.414+0000 7fe6c9b15640 1 -- 172.21.15.112:0/572825437 shutdown_connections 2024-03-29T10:59:55.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.414+0000 7fe6c9b15640 1 -- 172.21.15.112:0/572825437 >> 172.21.15.112:0/572825437 conn(0x7fe6c4066a00 msgr2=0x7fe6c4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:55.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.414+0000 7fe6c9b15640 1 -- 172.21.15.112:0/572825437 shutdown_connections 2024-03-29T10:59:55.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.414+0000 7fe6c9b15640 1 -- 172.21.15.112:0/572825437 wait complete. 2024-03-29T10:59:55.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.415+0000 7fe6c9b15640 10 monclient: build_initial_monmap 2024-03-29T10:59:55.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.415+0000 7fe6c9b15640 10 monclient: monmap: 2024-03-29T10:59:55.403 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T10:59:55.403 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:55.403 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T10:59:55.403 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T10:59:55.403 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:55.403 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:55.403 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T10:59:55.403 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T10:59:55.403 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T10:59:55.403 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:55.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.416+0000 7fe6c9b15640 1 librados: starting msgr at 2024-03-29T10:59:55.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.416+0000 7fe6c9b15640 1 librados: starting objecter 2024-03-29T10:59:55.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.416+0000 7fe6c9b15640 1 Processor -- start 2024-03-29T10:59:55.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.416+0000 7fe6c9b15640 1 -- start start 2024-03-29T10:59:55.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.416+0000 7fe6c9b15640 1 librados: setting wanted keys 2024-03-29T10:59:55.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.416+0000 7fe6c9b15640 1 librados: calling monclient init 2024-03-29T10:59:55.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.416+0000 7fe6c9b15640 10 monclient: init 2024-03-29T10:59:55.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.417+0000 7fe6c9b15640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:55.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.418+0000 7fe6c9b15640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:59:55.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.418+0000 7fe6c9b15640 10 monclient(hunting): picked mon.noname-c con 0x7fe6c40b17d0 addr v1:172.21.15.112:6790/0 2024-03-29T10:59:55.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.418+0000 7fe6c9b15640 10 monclient(hunting): picked mon.noname-b con 0x7fe6c41422b0 addr v1:172.21.15.118:6789/0 2024-03-29T10:59:55.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.418+0000 7fe6c9b15640 10 monclient(hunting): picked mon.noname-a con 0x7fe6c4142690 addr v1:172.21.15.112:6789/0 2024-03-29T10:59:55.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.418+0000 7fe6c9b15640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c40ac260 con 0x7fe6c4142690 2024-03-29T10:59:55.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.418+0000 7fe6c9b15640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c41322d0 con 0x7fe6c41422b0 2024-03-29T10:59:55.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.418+0000 7fe6c9b15640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c4132460 con 0x7fe6c40b17d0 2024-03-29T10:59:55.404 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.418+0000 7fe6c9b15640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:55.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.418+0000 7fe6c9b15640 10 monclient(hunting): authenticate will time out at 1350.722421s 2024-03-29T10:59:55.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.418+0000 7fe6c37fe640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe6c40b17d0 0x7fe6c41319f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:51300/0 (socket says 172.21.15.112:51300) 2024-03-29T10:59:55.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.418+0000 7fe6c37fe640 1 -- 172.21.15.112:0/2903964349 learned_addr learned my addr 172.21.15.112:0/2903964349 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T10:59:55.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.418+0000 7fe6abfff640 10 client.?.objecter ms_handle_connect 0x7fe6c4142690 2024-03-29T10:59:55.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.418+0000 7fe6abfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:55.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.418+0000 7fe6abfff640 1 -- 172.21.15.112:0/2903964349 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2614153833 0 0) 0x7fe6c40ac260 con 0x7fe6c4142690 2024-03-29T10:59:55.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.418+0000 7fe6abfff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:55.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.418+0000 7fe6abfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:55.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.418+0000 7fe6abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:55.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.418+0000 7fe6abfff640 10 monclient(hunting): my global_id is 4530 2024-03-29T10:59:55.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.418+0000 7fe6abfff640 1 -- 172.21.15.112:0/2903964349 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6b0004eb0 con 0x7fe6c4142690 2024-03-29T10:59:55.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.418+0000 7fe6abfff640 10 client.?.objecter ms_handle_connect 0x7fe6c40b17d0 2024-03-29T10:59:55.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.418+0000 7fe6abfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:55.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.419+0000 7fe6abfff640 1 -- 172.21.15.112:0/2903964349 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926380379 0 0) 0x7fe6c4132460 con 0x7fe6c40b17d0 2024-03-29T10:59:55.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.419+0000 7fe6abfff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:55.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.419+0000 7fe6abfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:55.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.419+0000 7fe6abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:55.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.419+0000 7fe6abfff640 10 monclient(hunting): my global_id is 4571 2024-03-29T10:59:55.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.419+0000 7fe6abfff640 1 -- 172.21.15.112:0/2903964349 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6c40ac260 con 0x7fe6c40b17d0 2024-03-29T10:59:55.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.419+0000 7fe6abfff640 1 -- 172.21.15.112:0/2903964349 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1283603863 0 0) 0x7fe6b0004eb0 con 0x7fe6c4142690 2024-03-29T10:59:55.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.419+0000 7fe6abfff640 1 -- 172.21.15.112:0/2903964349 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6c4132460 con 0x7fe6c4142690 2024-03-29T10:59:55.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.419+0000 7fe6abfff640 1 -- 172.21.15.112:0/2903964349 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe6bc001c70 con 0x7fe6c4142690 2024-03-29T10:59:55.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.419+0000 7fe6abfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:55.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.419+0000 7fe6abfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:59:55.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.419+0000 7fe6abfff640 10 monclient(hunting): dump: 2024-03-29T10:59:55.406 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:55.406 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:55.406 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:55.406 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:55.406 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:55.407 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:55.407 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:55.407 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:55.407 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:55.407 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:55.407 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.419+0000 7fe6abfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:55.407 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.420+0000 7fe6abfff640 1 -- 172.21.15.112:0/2903964349 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3548759365 0 0) 0x7fe6c40ac260 con 0x7fe6c40b17d0 2024-03-29T10:59:55.407 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.420+0000 7fe6abfff640 1 -- 172.21.15.112:0/2903964349 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6b0004eb0 con 0x7fe6c40b17d0 2024-03-29T10:59:55.407 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.420+0000 7fe6abfff640 1 -- 172.21.15.112:0/2903964349 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe6b40015d0 con 0x7fe6c40b17d0 2024-03-29T10:59:55.407 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.420+0000 7fe6abfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:55.407 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.420+0000 7fe6abfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:55.407 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.420+0000 7fe6abfff640 10 monclient(hunting): dump: 2024-03-29T10:59:55.407 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:55.407 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:55.407 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:55.408 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:55.408 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:55.408 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:55.408 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:55.408 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:55.408 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:55.408 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:55.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.420+0000 7fe6abfff640 10 client.?.objecter ms_handle_connect 0x7fe6c41422b0 2024-03-29T10:59:55.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.420+0000 7fe6abfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:55.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.420+0000 7fe6abfff640 1 -- 172.21.15.112:0/2903964349 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 541341316 0 0) 0x7fe6c4132460 con 0x7fe6c4142690 2024-03-29T10:59:55.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.420+0000 7fe6abfff640 1 -- 172.21.15.112:0/2903964349 >> v1:172.21.15.112:6790/0 conn(0x7fe6c40b17d0 legacy=0x7fe6c41319f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:55.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.420+0000 7fe6abfff640 1 -- 172.21.15.112:0/2903964349 >> v1:172.21.15.118:6789/0 conn(0x7fe6c41422b0 legacy=0x7fe6c40b08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:55.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.420+0000 7fe6abfff640 10 monclient: _finish_hunting 0 2024-03-29T10:59:55.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.420+0000 7fe6abfff640 1 monclient: found mon.a 2024-03-29T10:59:55.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.420+0000 7fe6abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:55.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.420+0000 7fe6abfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:59:55.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.420+0000 7fe6abfff640 1 -- 172.21.15.112:0/2903964349 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c41325f0 con 0x7fe6c4142690 2024-03-29T10:59:55.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.420+0000 7fe6abfff640 10 monclient: _finish_auth 0 2024-03-29T10:59:55.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.420+0000 7fe6abfff640 10 monclient: _check_auth_tickets 2024-03-29T10:59:55.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.420+0000 7fe6abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:55.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.420+0000 7fe6c9b15640 5 monclient: authenticate success, global_id 4530 2024-03-29T10:59:55.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.420+0000 7fe6c9b15640 10 monclient: _renew_subs 2024-03-29T10:59:55.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.420+0000 7fe6c9b15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:59:55.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.420+0000 7fe6c9b15640 1 -- 172.21.15.112:0/2903964349 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6c4130f40 con 0x7fe6c4142690 2024-03-29T10:59:55.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.421+0000 7fe6c9b15640 10 client.4530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:55.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.421+0000 7fe6c9b15640 10 monclient: _renew_subs 2024-03-29T10:59:55.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.421+0000 7fe6c9b15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:59:55.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.421+0000 7fe6c9b15640 1 -- 172.21.15.112:0/2903964349 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6c40b0ff0 con 0x7fe6c4142690 2024-03-29T10:59:55.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.421+0000 7fe6c9b15640 1 librados: init done 2024-03-29T10:59:55.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.421+0000 7fe6abfff640 1 -- 172.21.15.112:0/2903964349 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6bc003990 con 0x7fe6c4142690 2024-03-29T10:59:55.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.421+0000 7fe6abfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:55.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.421+0000 7fe6abfff640 1 -- 172.21.15.112:0/2903964349 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fe6bc003ca0 con 0x7fe6c4142690 2024-03-29T10:59:55.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.421+0000 7fe6c1ffb640 4 set_mon_vals no callback set 2024-03-29T10:59:55.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.421+0000 7fe6abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:55.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.421+0000 7fe6abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:55.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.421+0000 7fe6abfff640 10 monclient: dump: 2024-03-29T10:59:55.410 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T10:59:55.410 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T10:59:55.410 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:55.410 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T10:59:55.410 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:55.410 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T10:59:55.410 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T10:59:55.410 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T10:59:55.410 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T10:59:55.410 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T10:59:55.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.422+0000 7fe6c9b15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:55.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.422+0000 7fe6c9b15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:55.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.422+0000 7fe6c9b15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:59:55.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.422+0000 7fe6c9b15640 1 -- 172.21.15.112:0/2903964349 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6a4005080 con 0x7fe6c4142690 2024-03-29T10:59:55.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.423+0000 7fe6abfff640 1 -- 172.21.15.112:0/2903964349 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fe6bc019c70 con 0x7fe6c4142690 2024-03-29T10:59:55.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.424+0000 7fe6abfff640 1 -- 172.21.15.112:0/2903964349 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7fe6bc083b60 con 0x7fe6c4142690 2024-03-29T10:59:55.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.424+0000 7fe6abfff640 10 client.4530.objecter ms_dispatch 0x7fe6c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:55.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.424+0000 7fe6abfff640 3 client.4530.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:55.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.424+0000 7fe6c37fe640 10 monclient: get_auth_request con 0x7fe6b006de90 auth_method 0 2024-03-29T10:59:55.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.424+0000 7fe6abfff640 3 client.4530.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:55.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.424+0000 7fe6abfff640 20 client.4530.objecter dump_active .. 0 homeless 2024-03-29T10:59:55.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.425+0000 7fe6abfff640 10 client.4530.objecter ms_handle_connect 0x7fe6b006de90 2024-03-29T10:59:55.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.434+0000 7fe6abfff640 1 -- 172.21.15.112:0/2903964349 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe6bc04f440 con 0x7fe6c4142690 2024-03-29T10:59:55.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.434+0000 7fe6abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:55.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.434+0000 7fe6abfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:55.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.798+0000 7fe6c9b15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:55.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.798+0000 7fe6c9b15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:55.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.798+0000 7fe6c9b15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T10:59:55.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.798+0000 7fe6c9b15640 1 -- 172.21.15.112:0/2903964349 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6a4005330 con 0x7fe6c4142690 2024-03-29T10:59:55.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.799+0000 7fe6abfff640 1 -- 172.21.15.112:0/2903964349 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7fe6bc051370 con 0x7fe6c4142690 2024-03-29T10:59:55.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.799+0000 7fe6abfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:55.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.799+0000 7fe6abfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:55.780 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T10:59:55.780 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:55.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.803+0000 7fe6c9b15640 10 librados: watch_flush enter 2024-03-29T10:59:55.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.803+0000 7fe6c9b15640 10 librados: watch_flush exit 2024-03-29T10:59:55.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.803+0000 7fe6c9b15640 20 client.4530.objecter shutdown clearing up homeless session... 2024-03-29T10:59:55.784 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.803+0000 7fe6c9b15640 10 client.4530.objecter successfully canceled tick 2024-03-29T10:59:55.784 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.803+0000 7fe6c9b15640 1 -- 172.21.15.112:0/2903964349 >> v1:172.21.15.118:6816/3181073408 conn(0x7fe6b006de90 legacy=0x7fe6b0070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:55.784 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.803+0000 7fe6c9b15640 10 monclient: shutdown 2024-03-29T10:59:55.784 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.803+0000 7fe6c9b15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:55.784 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.803+0000 7fe6c9b15640 1 -- 172.21.15.112:0/2903964349 >> v1:172.21.15.112:6789/0 conn(0x7fe6c4142690 legacy=0x7fe6c41dac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:55.784 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.803+0000 7fe6c9b15640 1 -- 172.21.15.112:0/2903964349 shutdown_connections 2024-03-29T10:59:55.784 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.803+0000 7fe6c9b15640 1 -- 172.21.15.112:0/2903964349 >> 172.21.15.112:0/2903964349 conn(0x7fe6c4066a00 msgr2=0x7fe6c413f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:55.785 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.804+0000 7fe6c9b15640 1 -- 172.21.15.112:0/2903964349 shutdown_connections 2024-03-29T10:59:55.785 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.804+0000 7fe6c9b15640 1 -- 172.21.15.112:0/2903964349 wait complete. 2024-03-29T10:59:55.785 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T10:59:55.804+0000 7fe6c9b15640 1 librados: shutdown 2024-03-29T10:59:55.799 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:02.605 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:02.751 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.770+0000 7f254480e640 10 monclient: get_monmap_and_config 2024-03-29T11:00:02.751 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.770+0000 7f254480e640 10 monclient: build_initial_monmap 2024-03-29T11:00:02.751 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.770+0000 7f254480e640 10 monclient: monmap: 2024-03-29T11:00:02.752 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:00:02.752 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:02.752 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T11:00:02.772135+0000 2024-03-29T11:00:02.752 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T11:00:02.772135+0000 2024-03-29T11:00:02.752 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:02.752 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:02.752 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:00:02.752 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T11:00:02.752 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T11:00:02.752 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:02.754 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.771+0000 7f254480e640 1 Processor -- start 2024-03-29T11:00:02.754 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.771+0000 7f254480e640 1 -- start start 2024-03-29T11:00:02.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.771+0000 7f254480e640 10 monclient: init 2024-03-29T11:00:02.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.772+0000 7f254480e640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:02.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.772+0000 7f254480e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:00:02.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.772+0000 7f254480e640 10 monclient(hunting): picked mon.noname-c con 0x7f25401457f0 addr v1:172.21.15.118:6789/0 2024-03-29T11:00:02.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.772+0000 7f254480e640 10 monclient(hunting): picked mon.noname-b con 0x7f254013f650 addr v1:172.21.15.112:6790/0 2024-03-29T11:00:02.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.772+0000 7f254480e640 10 monclient(hunting): picked mon.noname-a con 0x7f25400a9d70 addr v1:172.21.15.112:6789/0 2024-03-29T11:00:02.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.772+0000 7f254480e640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25400aaef0 con 0x7f25400a9d70 2024-03-29T11:00:02.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.772+0000 7f254480e640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2540145bd0 con 0x7f25401457f0 2024-03-29T11:00:02.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.772+0000 7f254480e640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2540145eb0 con 0x7f254013f650 2024-03-29T11:00:02.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.772+0000 7f254480e640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:02.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.772+0000 7f254480e640 10 monclient(hunting): authenticate will time out at 1358.077664s 2024-03-29T11:00:02.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.772+0000 7f253dd9b640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f254013f650 0x7f25400b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:39990/0 (socket says 172.21.15.112:39990) 2024-03-29T11:00:02.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.772+0000 7f253dd9b640 1 -- 172.21.15.112:0/1111370611 learned_addr learned my addr 172.21.15.112:0/1111370611 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:00:02.756 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.772+0000 7f253d59a640 1 -- 172.21.15.112:0/1111370611 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1471838622 0 0) 0x7f2540145eb0 con 0x7f254013f650 2024-03-29T11:00:02.756 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.772+0000 7f253d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:00:02.756 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.772+0000 7f253d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:02.756 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.772+0000 7f253d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:02.756 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.772+0000 7f253d59a640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:00:02.756 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.772+0000 7f253d59a640 1 -- 172.21.15.112:0/1111370611 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f252c003520 con 0x7f254013f650 2024-03-29T11:00:02.756 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.773+0000 7f253d59a640 1 -- 172.21.15.112:0/1111370611 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 814162485 0 0) 0x7f252c003520 con 0x7f254013f650 2024-03-29T11:00:02.756 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.773+0000 7f253d59a640 1 -- 172.21.15.112:0/1111370611 >> v1:172.21.15.118:6789/0 conn(0x7f25401457f0 legacy=0x7f254013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:02.756 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.773+0000 7f253d59a640 1 -- 172.21.15.112:0/1111370611 >> v1:172.21.15.112:6789/0 conn(0x7f25400a9d70 legacy=0x7f254014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:00:02.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.774+0000 7f253d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:00:02.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.774+0000 7f253d59a640 1 monclient: found mon.noname-b 2024-03-29T11:00:02.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.774+0000 7f253d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:02.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.774+0000 7f253d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T11:00:02.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.774+0000 7f253d59a640 1 -- 172.21.15.112:0/1111370611 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2540146340 con 0x7f254013f650 2024-03-29T11:00:02.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.774+0000 7f253d59a640 10 monclient: _finish_auth 0 2024-03-29T11:00:02.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.774+0000 7f253d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:00:02.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.774+0000 7f253d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:02.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.774+0000 7f253d59a640 1 -- 172.21.15.112:0/1111370611 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f2528001690 con 0x7f254013f650 2024-03-29T11:00:02.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.774+0000 7f254480e640 5 monclient: authenticate success, global_id 4574 2024-03-29T11:00:02.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.774+0000 7f254480e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:02.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.774+0000 7f253d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:02.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.774+0000 7f253d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:02.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.774+0000 7f253d59a640 10 monclient: dump: 2024-03-29T11:00:02.762 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:02.762 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:02.762 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:02.762 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:02.762 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:02.762 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:02.763 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:02.763 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:02.763 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:02.763 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:02.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.774+0000 7f254480e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:02.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.775+0000 7f253d59a640 1 -- 172.21.15.112:0/1111370611 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2528001960 con 0x7f254013f650 2024-03-29T11:00:02.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.775+0000 7f253d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:02.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.775+0000 7f253d59a640 1 -- 172.21.15.112:0/1111370611 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f2528001cf0 con 0x7f254013f650 2024-03-29T11:00:02.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.775+0000 7f253d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:02.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.775+0000 7f253d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:02.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.775+0000 7f253d59a640 10 monclient: dump: 2024-03-29T11:00:02.763 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:02.763 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:02.764 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:02.764 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:02.764 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:02.764 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:02.764 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:02.764 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:02.764 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:02.764 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:02.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.775+0000 7f254480e640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:02.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.775+0000 7f254480e640 4 set_mon_vals no callback set 2024-03-29T11:00:02.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.776+0000 7f254480e640 10 monclient: shutdown 2024-03-29T11:00:02.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.776+0000 7f254480e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:02.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.776+0000 7f254480e640 1 -- 172.21.15.112:0/1111370611 >> v1:172.21.15.112:6790/0 conn(0x7f254013f650 legacy=0x7f25400b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:02.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.776+0000 7f254480e640 1 -- 172.21.15.112:0/1111370611 shutdown_connections 2024-03-29T11:00:02.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.776+0000 7f254480e640 1 -- 172.21.15.112:0/1111370611 >> 172.21.15.112:0/1111370611 conn(0x7f2540066a00 msgr2=0x7f2540066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:02.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.776+0000 7f254480e640 1 -- 172.21.15.112:0/1111370611 shutdown_connections 2024-03-29T11:00:02.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.776+0000 7f254480e640 1 -- 172.21.15.112:0/1111370611 wait complete. 2024-03-29T11:00:02.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.777+0000 7f254480e640 10 monclient: build_initial_monmap 2024-03-29T11:00:02.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.777+0000 7f254480e640 10 monclient: monmap: 2024-03-29T11:00:02.765 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:00:02.765 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:02.765 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T11:00:02.765 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T11:00:02.765 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:02.765 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:02.765 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:00:02.765 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T11:00:02.765 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T11:00:02.766 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:02.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.778+0000 7f254480e640 1 librados: starting msgr at 2024-03-29T11:00:02.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.778+0000 7f254480e640 1 librados: starting objecter 2024-03-29T11:00:02.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.778+0000 7f254480e640 1 Processor -- start 2024-03-29T11:00:02.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.778+0000 7f254480e640 1 -- start start 2024-03-29T11:00:02.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.778+0000 7f254480e640 1 librados: setting wanted keys 2024-03-29T11:00:02.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.778+0000 7f254480e640 1 librados: calling monclient init 2024-03-29T11:00:02.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.778+0000 7f254480e640 10 monclient: init 2024-03-29T11:00:02.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.778+0000 7f254480e640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:02.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.778+0000 7f254480e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:00:02.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.778+0000 7f254480e640 10 monclient(hunting): picked mon.noname-a con 0x7f25400a9d70 addr v1:172.21.15.112:6789/0 2024-03-29T11:00:02.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.779+0000 7f254480e640 10 monclient(hunting): picked mon.noname-c con 0x7f254013f650 addr v1:172.21.15.112:6790/0 2024-03-29T11:00:02.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.779+0000 7f253dd9b640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f254013f650 0x7f25401dab20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:40154/0 (socket says 172.21.15.112:40154) 2024-03-29T11:00:02.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.779+0000 7f253dd9b640 1 -- 172.21.15.112:0/2247755496 learned_addr learned my addr 172.21.15.112:0/2247755496 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:00:02.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.779+0000 7f254480e640 10 monclient(hunting): picked mon.noname-b con 0x7f25401457f0 addr v1:172.21.15.118:6789/0 2024-03-29T11:00:02.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.779+0000 7f254480e640 1 -- 172.21.15.112:0/2247755496 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25400ab6a0 con 0x7f25400a9d70 2024-03-29T11:00:02.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.779+0000 7f2536ffd640 10 client.?.objecter ms_handle_connect 0x7f254013f650 2024-03-29T11:00:02.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.779+0000 7f2536ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:02.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.779+0000 7f254480e640 1 -- 172.21.15.112:0/2247755496 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2540140100 con 0x7f25401457f0 2024-03-29T11:00:02.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.779+0000 7f254480e640 1 -- 172.21.15.112:0/2247755496 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2540140290 con 0x7f254013f650 2024-03-29T11:00:02.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.779+0000 7f254480e640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:02.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.779+0000 7f254480e640 10 monclient(hunting): authenticate will time out at 1358.084438s 2024-03-29T11:00:02.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.779+0000 7f2536ffd640 1 -- 172.21.15.112:0/2247755496 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 176970071 0 0) 0x7f2540140290 con 0x7f254013f650 2024-03-29T11:00:02.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.779+0000 7f2536ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:02.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.779+0000 7f2536ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:02.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.779+0000 7f2536ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:02.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.779+0000 7f2536ffd640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:00:02.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.779+0000 7f2536ffd640 1 -- 172.21.15.112:0/2247755496 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2520004eb0 con 0x7f254013f650 2024-03-29T11:00:02.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.780+0000 7f2536ffd640 1 -- 172.21.15.112:0/2247755496 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2623634341 0 0) 0x7f2520004eb0 con 0x7f254013f650 2024-03-29T11:00:02.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.780+0000 7f2536ffd640 1 -- 172.21.15.112:0/2247755496 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2540140290 con 0x7f254013f650 2024-03-29T11:00:02.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.780+0000 7f2536ffd640 1 -- 172.21.15.112:0/2247755496 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f2528001650 con 0x7f254013f650 2024-03-29T11:00:02.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.780+0000 7f2536ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:02.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.780+0000 7f2536ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:02.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.780+0000 7f2536ffd640 10 monclient(hunting): dump: 2024-03-29T11:00:02.768 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:02.768 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:02.768 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:02.768 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:02.768 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:02.768 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:02.768 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:02.768 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:02.768 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:02.769 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:02.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.780+0000 7f2536ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:02.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.780+0000 7f2536ffd640 1 -- 172.21.15.112:0/2247755496 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3393794256 0 0) 0x7f2540140290 con 0x7f254013f650 2024-03-29T11:00:02.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.780+0000 7f2536ffd640 1 -- 172.21.15.112:0/2247755496 >> v1:172.21.15.118:6789/0 conn(0x7f25401457f0 legacy=0x7f25401db230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:02.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.780+0000 7f2536ffd640 1 -- 172.21.15.112:0/2247755496 >> v1:172.21.15.112:6789/0 conn(0x7f25400a9d70 legacy=0x7f25400b0d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:00:02.771 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.782+0000 7f2536ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:00:02.771 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.782+0000 7f2536ffd640 1 monclient: found mon.c 2024-03-29T11:00:02.771 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.782+0000 7f2536ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:02.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.782+0000 7f2536ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:02.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.782+0000 7f2536ffd640 1 -- 172.21.15.112:0/2247755496 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25400ac280 con 0x7f254013f650 2024-03-29T11:00:02.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.782+0000 7f2536ffd640 10 monclient: _finish_auth 0 2024-03-29T11:00:02.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.782+0000 7f2536ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:00:02.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.782+0000 7f2536ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:02.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.782+0000 7f254480e640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:00:02.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.782+0000 7f254480e640 10 monclient: _renew_subs 2024-03-29T11:00:02.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.782+0000 7f254480e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:02.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.782+0000 7f254480e640 1 -- 172.21.15.112:0/2247755496 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25400ac4d0 con 0x7f254013f650 2024-03-29T11:00:02.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.782+0000 7f254480e640 10 client.4577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:02.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.782+0000 7f254480e640 10 monclient: _renew_subs 2024-03-29T11:00:02.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.782+0000 7f254480e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:02.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.782+0000 7f254480e640 1 -- 172.21.15.112:0/2247755496 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25400b1680 con 0x7f254013f650 2024-03-29T11:00:02.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.782+0000 7f254480e640 1 librados: init done 2024-03-29T11:00:02.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.783+0000 7f2536ffd640 1 -- 172.21.15.112:0/2247755496 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2528004c30 con 0x7f254013f650 2024-03-29T11:00:02.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.783+0000 7f2536ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:02.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.783+0000 7f2536ffd640 1 -- 172.21.15.112:0/2247755496 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f2528004f40 con 0x7f254013f650 2024-03-29T11:00:02.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.783+0000 7f2536ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:02.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.783+0000 7f2536ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:02.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.783+0000 7f2536ffd640 10 monclient: dump: 2024-03-29T11:00:02.773 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:02.773 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:02.773 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:02.773 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:02.773 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:02.773 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:02.773 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:02.774 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:02.774 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:02.774 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:02.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.783+0000 7f253cd99640 4 set_mon_vals no callback set 2024-03-29T11:00:02.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.784+0000 7f254480e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:02.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.785+0000 7f254480e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:02.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.785+0000 7f254480e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:02.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.785+0000 7f2536ffd640 1 -- 172.21.15.112:0/2247755496 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f252801af10 con 0x7f254013f650 2024-03-29T11:00:02.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.785+0000 7f254480e640 1 -- 172.21.15.112:0/2247755496 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2524005080 con 0x7f254013f650 2024-03-29T11:00:02.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.785+0000 7f2536ffd640 1 -- 172.21.15.112:0/2247755496 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f2528084cd0 con 0x7f254013f650 2024-03-29T11:00:02.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.785+0000 7f2536ffd640 10 client.4577.objecter ms_dispatch 0x7f2540000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:02.774 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.785+0000 7f2536ffd640 3 client.4577.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:02.775 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.785+0000 7f2536ffd640 3 client.4577.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:02.775 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.785+0000 7f2536ffd640 20 client.4577.objecter dump_active .. 0 homeless 2024-03-29T11:00:02.775 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.785+0000 7f253e59c640 10 monclient: get_auth_request con 0x7f252006dc70 auth_method 0 2024-03-29T11:00:02.775 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.786+0000 7f2536ffd640 10 client.4577.objecter ms_handle_connect 0x7f252006dc70 2024-03-29T11:00:02.775 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.793+0000 7f2536ffd640 1 -- 172.21.15.112:0/2247755496 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f25280505b0 con 0x7f254013f650 2024-03-29T11:00:02.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.793+0000 7f2536ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:02.776 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:02.793+0000 7f2536ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:03.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:03.096+0000 7f254480e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:03.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:03.096+0000 7f254480e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:03.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:03.096+0000 7f254480e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:03.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:03.096+0000 7f254480e640 1 -- 172.21.15.112:0/2247755496 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2524005330 con 0x7f254013f650 2024-03-29T11:00:03.079 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:03.097+0000 7f2536ffd640 1 -- 172.21.15.112:0/2247755496 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f2528001210 con 0x7f254013f650 2024-03-29T11:00:03.079 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:03.097+0000 7f2536ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:03.079 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:03.097+0000 7f2536ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:03.079 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T11:00:03.079 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:03.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:03.100+0000 7f254480e640 10 librados: watch_flush enter 2024-03-29T11:00:03.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:03.100+0000 7f254480e640 10 librados: watch_flush exit 2024-03-29T11:00:03.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:03.100+0000 7f254480e640 20 client.4577.objecter shutdown clearing up homeless session... 2024-03-29T11:00:03.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:03.100+0000 7f254480e640 10 client.4577.objecter successfully canceled tick 2024-03-29T11:00:03.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:03.100+0000 7f254480e640 1 -- 172.21.15.112:0/2247755496 >> v1:172.21.15.118:6816/3181073408 conn(0x7f252006dc70 legacy=0x7f2520070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:03.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:03.100+0000 7f254480e640 10 monclient: shutdown 2024-03-29T11:00:03.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:03.100+0000 7f254480e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:03.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:03.100+0000 7f254480e640 1 -- 172.21.15.112:0/2247755496 >> v1:172.21.15.112:6790/0 conn(0x7f254013f650 legacy=0x7f25401dab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:03.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:03.100+0000 7f254480e640 1 -- 172.21.15.112:0/2247755496 shutdown_connections 2024-03-29T11:00:03.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:03.100+0000 7f254480e640 1 -- 172.21.15.112:0/2247755496 >> 172.21.15.112:0/2247755496 conn(0x7f2540066a00 msgr2=0x7f254013b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:03.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:03.100+0000 7f254480e640 1 -- 172.21.15.112:0/2247755496 shutdown_connections 2024-03-29T11:00:03.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:03.100+0000 7f254480e640 1 -- 172.21.15.112:0/2247755496 wait complete. 2024-03-29T11:00:03.082 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:03.100+0000 7f254480e640 1 librados: shutdown 2024-03-29T11:00:03.094 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:09.902 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:10.054 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.073+0000 7fa7e9c82640 10 monclient: get_monmap_and_config 2024-03-29T11:00:10.054 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.073+0000 7fa7e9c82640 10 monclient: build_initial_monmap 2024-03-29T11:00:10.054 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.073+0000 7fa7e9c82640 10 monclient: monmap: 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T11:00:10.074387+0000 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T11:00:10.074387+0000 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.074+0000 7fa7e9c82640 1 Processor -- start 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.074+0000 7fa7e9c82640 1 -- start start 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.074+0000 7fa7e9c82640 10 monclient: init 2024-03-29T11:00:10.056 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.075+0000 7fa7e9c82640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:10.056 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.075+0000 7fa7e9c82640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:00:10.058 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.078+0000 7fa7e9c82640 10 monclient(hunting): picked mon.noname-c con 0x7fa7e40a9750 addr v1:172.21.15.118:6789/0 2024-03-29T11:00:10.058 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.078+0000 7fa7e9c82640 10 monclient(hunting): picked mon.noname-b con 0x7fa7e40adca0 addr v1:172.21.15.112:6790/0 2024-03-29T11:00:10.058 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.078+0000 7fa7e9c82640 10 monclient(hunting): picked mon.noname-a con 0x7fa7e4146a80 addr v1:172.21.15.112:6789/0 2024-03-29T11:00:10.058 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.078+0000 7fa7e9c82640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e40abdc0 con 0x7fa7e4146a80 2024-03-29T11:00:10.059 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.078+0000 7fa7e9c82640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e40b0070 con 0x7fa7e40a9750 2024-03-29T11:00:10.059 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.078+0000 7fa7e9c82640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e40af880 con 0x7fa7e40adca0 2024-03-29T11:00:10.059 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.078+0000 7fa7e9c82640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:10.059 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.078+0000 7fa7e9c82640 10 monclient(hunting): authenticate will time out at 1365.382331s 2024-03-29T11:00:10.059 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.078+0000 7fa7e3fff640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fa7e40adca0 0x7fa7e4146370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:44712/0 (socket says 172.21.15.112:44712) 2024-03-29T11:00:10.059 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.078+0000 7fa7e3fff640 1 -- 172.21.15.112:0/3122644535 learned_addr learned my addr 172.21.15.112:0/3122644535 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:00:10.061 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.081+0000 7fa7e37fe640 1 -- 172.21.15.112:0/3122644535 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025761176 0 0) 0x7fa7e40af880 con 0x7fa7e40adca0 2024-03-29T11:00:10.061 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.081+0000 7fa7e37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:00:10.061 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.081+0000 7fa7e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:10.061 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.081+0000 7fa7e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:10.061 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.081+0000 7fa7e37fe640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:00:10.061 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.081+0000 7fa7e37fe640 1 -- 172.21.15.112:0/3122644535 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7d0003520 con 0x7fa7e40adca0 2024-03-29T11:00:10.061 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.081+0000 7fa7e37fe640 1 -- 172.21.15.112:0/3122644535 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296823227 0 0) 0x7fa7e40b0070 con 0x7fa7e40a9750 2024-03-29T11:00:10.062 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.081+0000 7fa7e37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:00:10.062 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.081+0000 7fa7e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:10.062 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.081+0000 7fa7e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:10.062 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.081+0000 7fa7e37fe640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:00:10.062 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.081+0000 7fa7e37fe640 1 -- 172.21.15.112:0/3122644535 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7e40af880 con 0x7fa7e40a9750 2024-03-29T11:00:10.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.082+0000 7fa7e37fe640 1 -- 172.21.15.112:0/3122644535 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2714100185 0 0) 0x7fa7e40af880 con 0x7fa7e40a9750 2024-03-29T11:00:10.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.082+0000 7fa7e37fe640 1 -- 172.21.15.112:0/3122644535 >> v1:172.21.15.112:6790/0 conn(0x7fa7e40adca0 legacy=0x7fa7e4146370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:10.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.082+0000 7fa7e37fe640 1 -- 172.21.15.112:0/3122644535 >> v1:172.21.15.112:6789/0 conn(0x7fa7e4146a80 legacy=0x7fa7e4151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:00:10.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.082+0000 7fa7e37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:00:10.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.082+0000 7fa7e37fe640 1 monclient: found mon.noname-c 2024-03-29T11:00:10.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.082+0000 7fa7e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:10.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.082+0000 7fa7e37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T11:00:10.065 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.082+0000 7fa7e37fe640 1 -- 172.21.15.112:0/3122644535 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e40ac030 con 0x7fa7e40a9750 2024-03-29T11:00:10.066 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.082+0000 7fa7e37fe640 10 monclient: _finish_auth 0 2024-03-29T11:00:10.066 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.082+0000 7fa7e37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:00:10.066 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.082+0000 7fa7e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:10.066 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.083+0000 7fa7e37fe640 1 -- 172.21.15.112:0/3122644535 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fa7d4001520 con 0x7fa7e40a9750 2024-03-29T11:00:10.066 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.083+0000 7fa7e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:10.066 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.083+0000 7fa7e37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:10.066 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.083+0000 7fa7e37fe640 10 monclient: dump: 2024-03-29T11:00:10.066 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:10.066 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:10.066 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:10.066 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:10.066 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:10.066 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:10.066 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:10.067 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:10.067 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:10.067 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:10.067 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.083+0000 7fa7e37fe640 1 -- 172.21.15.112:0/3122644535 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7d4001740 con 0x7fa7e40a9750 2024-03-29T11:00:10.067 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.083+0000 7fa7e37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:10.067 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.083+0000 7fa7e37fe640 1 -- 172.21.15.112:0/3122644535 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fa7d4001ad0 con 0x7fa7e40a9750 2024-03-29T11:00:10.067 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.083+0000 7fa7e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:10.067 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.083+0000 7fa7e37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:10.067 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.083+0000 7fa7e37fe640 10 monclient: dump: 2024-03-29T11:00:10.067 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:10.067 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:10.067 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:10.067 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:10.067 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:10.068 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:10.068 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:10.068 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:10.068 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:10.068 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:10.068 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.083+0000 7fa7e9c82640 5 monclient: authenticate success, global_id 4510 2024-03-29T11:00:10.068 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.083+0000 7fa7e9c82640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:10.068 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.083+0000 7fa7e9c82640 4 set_mon_vals no callback set 2024-03-29T11:00:10.068 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.084+0000 7fa7e9c82640 10 monclient: shutdown 2024-03-29T11:00:10.068 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.084+0000 7fa7e9c82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:10.068 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.084+0000 7fa7e9c82640 1 -- 172.21.15.112:0/3122644535 >> v1:172.21.15.118:6789/0 conn(0x7fa7e40a9750 legacy=0x7fa7e4143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:10.068 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.084+0000 7fa7e9c82640 1 -- 172.21.15.112:0/3122644535 shutdown_connections 2024-03-29T11:00:10.068 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.084+0000 7fa7e9c82640 1 -- 172.21.15.112:0/3122644535 >> 172.21.15.112:0/3122644535 conn(0x7fa7e40673e0 msgr2=0x7fa7e40b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:10.068 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.085+0000 7fa7e9c82640 1 -- 172.21.15.112:0/3122644535 shutdown_connections 2024-03-29T11:00:10.069 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.085+0000 7fa7e9c82640 1 -- 172.21.15.112:0/3122644535 wait complete. 2024-03-29T11:00:10.071 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.086+0000 7fa7e9c82640 10 monclient: build_initial_monmap 2024-03-29T11:00:10.071 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.086+0000 7fa7e9c82640 10 monclient: monmap: 2024-03-29T11:00:10.071 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:00:10.071 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:10.071 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T11:00:10.071 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T11:00:10.071 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:10.071 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:10.071 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:00:10.071 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T11:00:10.072 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T11:00:10.072 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:10.072 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.087+0000 7fa7e9c82640 1 librados: starting msgr at 2024-03-29T11:00:10.072 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.087+0000 7fa7e9c82640 1 librados: starting objecter 2024-03-29T11:00:10.072 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.087+0000 7fa7e9c82640 1 Processor -- start 2024-03-29T11:00:10.072 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.087+0000 7fa7e9c82640 1 -- start start 2024-03-29T11:00:10.072 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.087+0000 7fa7e9c82640 1 librados: setting wanted keys 2024-03-29T11:00:10.072 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.087+0000 7fa7e9c82640 1 librados: calling monclient init 2024-03-29T11:00:10.072 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.087+0000 7fa7e9c82640 10 monclient: init 2024-03-29T11:00:10.072 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.088+0000 7fa7e9c82640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:10.072 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.088+0000 7fa7e9c82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:00:10.072 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.088+0000 7fa7e9c82640 10 monclient(hunting): picked mon.noname-a con 0x7fa7e40a9750 addr v1:172.21.15.112:6789/0 2024-03-29T11:00:10.072 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.088+0000 7fa7e9c82640 10 monclient(hunting): picked mon.noname-b con 0x7fa7e40adca0 addr v1:172.21.15.118:6789/0 2024-03-29T11:00:10.072 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.088+0000 7fa7e9c82640 10 monclient(hunting): picked mon.noname-c con 0x7fa7e4146a80 addr v1:172.21.15.112:6790/0 2024-03-29T11:00:10.072 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.088+0000 7fa7e9c82640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e41db450 con 0x7fa7e40a9750 2024-03-29T11:00:10.073 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.088+0000 7fa7e9c82640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e40b1410 con 0x7fa7e40adca0 2024-03-29T11:00:10.073 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.088+0000 7fa7e9c82640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e40b15a0 con 0x7fa7e4146a80 2024-03-29T11:00:10.073 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.088+0000 7fa7e9c82640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:10.073 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.088+0000 7fa7e9c82640 10 monclient(hunting): authenticate will time out at 1365.393069s 2024-03-29T11:00:10.073 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.090+0000 7fa7e9481640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fa7e4146a80 0x7fa7e41e5df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:44948/0 (socket says 172.21.15.112:44948) 2024-03-29T11:00:10.073 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.090+0000 7fa7e9481640 1 -- 172.21.15.112:0/3655916418 learned_addr learned my addr 172.21.15.112:0/3655916418 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:00:10.074 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.092+0000 7fa7e17fa640 10 client.?.objecter ms_handle_connect 0x7fa7e4146a80 2024-03-29T11:00:10.074 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.092+0000 7fa7e17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:10.074 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.093+0000 7fa7e17fa640 1 -- 172.21.15.112:0/3655916418 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273409722 0 0) 0x7fa7e40b15a0 con 0x7fa7e4146a80 2024-03-29T11:00:10.074 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.093+0000 7fa7e17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:00:10.074 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.093+0000 7fa7e17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:10.074 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.093+0000 7fa7e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:10.074 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.093+0000 7fa7e17fa640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:00:10.075 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.093+0000 7fa7e17fa640 1 -- 172.21.15.112:0/3655916418 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7c4004eb0 con 0x7fa7e4146a80 2024-03-29T11:00:10.075 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.093+0000 7fa7e17fa640 10 client.?.objecter ms_handle_connect 0x7fa7e40a9750 2024-03-29T11:00:10.075 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.093+0000 7fa7e17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:10.075 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.093+0000 7fa7e17fa640 1 -- 172.21.15.112:0/3655916418 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3176953635 0 0) 0x7fa7c4004eb0 con 0x7fa7e4146a80 2024-03-29T11:00:10.075 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.093+0000 7fa7e17fa640 1 -- 172.21.15.112:0/3655916418 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7e40b15a0 con 0x7fa7e4146a80 2024-03-29T11:00:10.075 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.093+0000 7fa7e17fa640 1 -- 172.21.15.112:0/3655916418 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fa7dc001750 con 0x7fa7e4146a80 2024-03-29T11:00:10.075 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.093+0000 7fa7e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:10.075 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.093+0000 7fa7e17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:10.075 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.093+0000 7fa7e17fa640 10 monclient(hunting): dump: 2024-03-29T11:00:10.075 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:10.075 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:10.075 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:10.075 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:10.075 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:10.075 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:10.076 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:10.076 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:10.076 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:10.076 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:10.076 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:10.076 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e17fa640 10 client.?.objecter ms_handle_connect 0x7fa7e40adca0 2024-03-29T11:00:10.076 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:10.076 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e17fa640 1 -- 172.21.15.112:0/3655916418 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 847867444 0 0) 0x7fa7e41db450 con 0x7fa7e40a9750 2024-03-29T11:00:10.076 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:00:10.076 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:10.076 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:10.076 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e17fa640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:00:10.076 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e17fa640 1 -- 172.21.15.112:0/3655916418 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7c4004eb0 con 0x7fa7e40a9750 2024-03-29T11:00:10.076 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e17fa640 1 -- 172.21.15.112:0/3655916418 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2470974774 0 0) 0x7fa7e40b15a0 con 0x7fa7e4146a80 2024-03-29T11:00:10.076 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e17fa640 1 -- 172.21.15.112:0/3655916418 >> v1:172.21.15.118:6789/0 conn(0x7fa7e40adca0 legacy=0x7fa7e41db950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:10.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e17fa640 1 -- 172.21.15.112:0/3655916418 >> v1:172.21.15.112:6789/0 conn(0x7fa7e40a9750 legacy=0x7fa7e4145190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:10.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:00:10.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e17fa640 1 monclient: found mon.c 2024-03-29T11:00:10.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:10.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:10.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e17fa640 1 -- 172.21.15.112:0/3655916418 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e41458a0 con 0x7fa7e4146a80 2024-03-29T11:00:10.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e17fa640 10 monclient: _finish_auth 0 2024-03-29T11:00:10.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:00:10.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:10.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e9c82640 5 monclient: authenticate success, global_id 4583 2024-03-29T11:00:10.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e9c82640 10 monclient: _renew_subs 2024-03-29T11:00:10.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e9c82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:10.077 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e9c82640 1 -- 172.21.15.112:0/3655916418 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e41dc060 con 0x7fa7e4146a80 2024-03-29T11:00:10.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e9c82640 10 client.4583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:10.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e9c82640 10 monclient: _renew_subs 2024-03-29T11:00:10.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e9c82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:10.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e9c82640 1 -- 172.21.15.112:0/3655916418 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e41446d0 con 0x7fa7e4146a80 2024-03-29T11:00:10.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.094+0000 7fa7e9c82640 1 librados: init done 2024-03-29T11:00:10.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.096+0000 7fa7e9c82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:10.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.096+0000 7fa7e9c82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:10.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.096+0000 7fa7e9c82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:10.079 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.096+0000 7fa7e9c82640 1 -- 172.21.15.112:0/3655916418 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e404f730 con 0x7fa7e4146a80 2024-03-29T11:00:10.079 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.096+0000 7fa7e17fa640 1 -- 172.21.15.112:0/3655916418 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7dc001ce0 con 0x7fa7e4146a80 2024-03-29T11:00:10.079 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.096+0000 7fa7e17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:10.079 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.096+0000 7fa7e17fa640 1 -- 172.21.15.112:0/3655916418 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fa7dc004a00 con 0x7fa7e4146a80 2024-03-29T11:00:10.079 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.096+0000 7fa7e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:10.079 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.096+0000 7fa7e17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:10.079 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.096+0000 7fa7e17fa640 10 monclient: dump: 2024-03-29T11:00:10.079 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:10.079 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:10.079 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:10.079 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:10.079 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:10.079 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:10.079 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:10.079 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:10.079 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:10.080 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:10.080 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.097+0000 7fa7e27fc640 4 set_mon_vals no callback set 2024-03-29T11:00:10.080 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.099+0000 7fa7e17fa640 1 -- 172.21.15.112:0/3655916418 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fa7dc01aa10 con 0x7fa7e4146a80 2024-03-29T11:00:10.080 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.100+0000 7fa7e8c80640 10 monclient: get_auth_request con 0x7fa7c406dd00 auth_method 0 2024-03-29T11:00:10.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.101+0000 7fa7e17fa640 10 client.4583.objecter ms_handle_connect 0x7fa7c406dd00 2024-03-29T11:00:10.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.101+0000 7fa7e17fa640 1 -- 172.21.15.112:0/3655916418 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7fa7dc0547d0 con 0x7fa7e4146a80 2024-03-29T11:00:10.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.101+0000 7fa7e17fa640 10 client.4583.objecter ms_dispatch 0x7fa7e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:10.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.101+0000 7fa7e17fa640 3 client.4583.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:10.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.101+0000 7fa7e17fa640 3 client.4583.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:10.081 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.101+0000 7fa7e17fa640 20 client.4583.objecter dump_active .. 0 homeless 2024-03-29T11:00:10.089 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.110+0000 7fa7e17fa640 1 -- 172.21.15.112:0/3655916418 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa7dc050a70 con 0x7fa7e4146a80 2024-03-29T11:00:10.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.110+0000 7fa7e17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:10.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.110+0000 7fa7e17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:10.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.429+0000 7fa7e9c82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:10.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.429+0000 7fa7e9c82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:10.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.429+0000 7fa7e9c82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:10.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.429+0000 7fa7e9c82640 1 -- 172.21.15.112:0/3655916418 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7e41db640 con 0x7fa7e4146a80 2024-03-29T11:00:10.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.430+0000 7fa7e17fa640 1 -- 172.21.15.112:0/3655916418 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7fa7dc050a70 con 0x7fa7e4146a80 2024-03-29T11:00:10.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.430+0000 7fa7e17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:10.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.430+0000 7fa7e17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:10.411 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T11:00:10.411 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:10.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.433+0000 7fa7e9c82640 10 librados: watch_flush enter 2024-03-29T11:00:10.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.433+0000 7fa7e9c82640 10 librados: watch_flush exit 2024-03-29T11:00:10.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.433+0000 7fa7e9c82640 20 client.4583.objecter shutdown clearing up homeless session... 2024-03-29T11:00:10.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.433+0000 7fa7e9c82640 10 client.4583.objecter successfully canceled tick 2024-03-29T11:00:10.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.433+0000 7fa7e9c82640 1 -- 172.21.15.112:0/3655916418 >> v1:172.21.15.118:6816/3181073408 conn(0x7fa7c406dd00 legacy=0x7fa7c40701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:10.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.433+0000 7fa7e9c82640 10 monclient: shutdown 2024-03-29T11:00:10.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.433+0000 7fa7e9c82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:10.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.433+0000 7fa7e9c82640 1 -- 172.21.15.112:0/3655916418 >> v1:172.21.15.112:6790/0 conn(0x7fa7e4146a80 legacy=0x7fa7e41e5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:10.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.434+0000 7fa7e9c82640 1 -- 172.21.15.112:0/3655916418 shutdown_connections 2024-03-29T11:00:10.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.434+0000 7fa7e9c82640 1 -- 172.21.15.112:0/3655916418 >> 172.21.15.112:0/3655916418 conn(0x7fa7e40673e0 msgr2=0x7fa7e40aa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:10.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.434+0000 7fa7e9c82640 1 -- 172.21.15.112:0/3655916418 shutdown_connections 2024-03-29T11:00:10.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.434+0000 7fa7e9c82640 1 -- 172.21.15.112:0/3655916418 wait complete. 2024-03-29T11:00:10.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:10.434+0000 7fa7e9c82640 1 librados: shutdown 2024-03-29T11:00:10.430 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:17.236 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:17.389 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.409+0000 7f15761af640 10 monclient: get_monmap_and_config 2024-03-29T11:00:17.389 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.409+0000 7f15761af640 10 monclient: build_initial_monmap 2024-03-29T11:00:17.389 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.409+0000 7f15761af640 10 monclient: monmap: 2024-03-29T11:00:17.390 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:00:17.390 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:17.390 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T11:00:17.410416+0000 2024-03-29T11:00:17.390 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T11:00:17.410416+0000 2024-03-29T11:00:17.390 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:17.390 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:17.390 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:00:17.390 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T11:00:17.390 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T11:00:17.390 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:17.393 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.410+0000 7f15761af640 1 Processor -- start 2024-03-29T11:00:17.393 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.410+0000 7f15761af640 1 -- start start 2024-03-29T11:00:17.393 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.410+0000 7f15761af640 10 monclient: init 2024-03-29T11:00:17.393 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.411+0000 7f15761af640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:17.393 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.411+0000 7f15761af640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:00:17.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.411+0000 7f15761af640 10 monclient(hunting): picked mon.noname-a con 0x7f15700b1f60 addr v1:172.21.15.112:6789/0 2024-03-29T11:00:17.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.411+0000 7f15761af640 10 monclient(hunting): picked mon.noname-b con 0x7f1570071eb0 addr v1:172.21.15.112:6790/0 2024-03-29T11:00:17.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.411+0000 7f15761af640 10 monclient(hunting): picked mon.noname-c con 0x7f1570143d10 addr v1:172.21.15.118:6789/0 2024-03-29T11:00:17.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.411+0000 7f15761af640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f157006ef80 con 0x7f15700b1f60 2024-03-29T11:00:17.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.411+0000 7f15761af640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f157006f260 con 0x7f1570143d10 2024-03-29T11:00:17.395 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.414+0000 7f156effd640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f1570071eb0 0x7f1570146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41964/0 (socket says 172.21.15.112:41964) 2024-03-29T11:00:17.395 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.414+0000 7f156effd640 1 -- 172.21.15.112:0/970684467 learned_addr learned my addr 172.21.15.112:0/970684467 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:00:17.395 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.414+0000 7f15761af640 1 -- 172.21.15.112:0/970684467 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15700b40c0 con 0x7f1570071eb0 2024-03-29T11:00:17.395 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.414+0000 7f15761af640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:17.395 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.414+0000 7f15761af640 10 monclient(hunting): authenticate will time out at 1372.718989s 2024-03-29T11:00:17.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.415+0000 7f156e7fc640 1 -- 172.21.15.112:0/970684467 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666224628 0 0) 0x7f157006ef80 con 0x7f15700b1f60 2024-03-29T11:00:17.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.415+0000 7f156e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:17.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.415+0000 7f156e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:17.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.415+0000 7f156e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:17.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.415+0000 7f156e7fc640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:00:17.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.415+0000 7f156e7fc640 1 -- 172.21.15.112:0/970684467 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1558003520 con 0x7f15700b1f60 2024-03-29T11:00:17.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.416+0000 7f156e7fc640 1 -- 172.21.15.112:0/970684467 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1343328681 0 0) 0x7f15700b40c0 con 0x7f1570071eb0 2024-03-29T11:00:17.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.416+0000 7f156e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:17.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.416+0000 7f156e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:17.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.416+0000 7f156e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:17.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.416+0000 7f156e7fc640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:00:17.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.416+0000 7f156e7fc640 1 -- 172.21.15.112:0/970684467 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f157006ef80 con 0x7f1570071eb0 2024-03-29T11:00:17.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.416+0000 7f156e7fc640 1 -- 172.21.15.112:0/970684467 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 859672273 0 0) 0x7f157006ef80 con 0x7f1570071eb0 2024-03-29T11:00:17.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.416+0000 7f156e7fc640 1 -- 172.21.15.112:0/970684467 >> v1:172.21.15.118:6789/0 conn(0x7f1570143d10 legacy=0x7f1570146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:00:17.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.416+0000 7f156e7fc640 1 -- 172.21.15.112:0/970684467 >> v1:172.21.15.112:6789/0 conn(0x7f15700b1f60 legacy=0x7f1570143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:17.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.416+0000 7f156e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:00:17.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.416+0000 7f156e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:00:17.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.416+0000 7f156e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:17.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.416+0000 7f156e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T11:00:17.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.416+0000 7f156e7fc640 1 -- 172.21.15.112:0/970684467 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1570154d60 con 0x7f1570071eb0 2024-03-29T11:00:17.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.416+0000 7f156e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:00:17.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.416+0000 7f156e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:00:17.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.416+0000 7f156e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:17.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.416+0000 7f156e7fc640 1 -- 172.21.15.112:0/970684467 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1564001670 con 0x7f1570071eb0 2024-03-29T11:00:17.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.416+0000 7f156e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:17.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.416+0000 7f156e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:17.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.416+0000 7f156e7fc640 10 monclient: dump: 2024-03-29T11:00:17.400 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:17.400 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:17.400 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:17.400 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:17.400 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:17.401 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:17.401 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:17.401 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:17.401 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:17.401 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:17.401 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.417+0000 7f15761af640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:00:17.401 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.417+0000 7f15761af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:17.401 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.417+0000 7f156e7fc640 1 -- 172.21.15.112:0/970684467 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15640017d0 con 0x7f1570071eb0 2024-03-29T11:00:17.401 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.417+0000 7f156e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:17.401 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.417+0000 7f156e7fc640 1 -- 172.21.15.112:0/970684467 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1564001ba0 con 0x7f1570071eb0 2024-03-29T11:00:17.401 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.417+0000 7f156e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:17.401 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.417+0000 7f156e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:17.401 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.417+0000 7f156e7fc640 10 monclient: dump: 2024-03-29T11:00:17.401 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:17.402 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:17.402 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:17.402 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:17.402 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:17.402 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:17.402 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:17.402 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:17.402 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:17.402 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:17.402 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.417+0000 7f15761af640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:17.402 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.417+0000 7f15761af640 4 set_mon_vals no callback set 2024-03-29T11:00:17.402 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.418+0000 7f15761af640 10 monclient: shutdown 2024-03-29T11:00:17.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.418+0000 7f15761af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:17.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.418+0000 7f15761af640 1 -- 172.21.15.112:0/970684467 >> v1:172.21.15.112:6790/0 conn(0x7f1570071eb0 legacy=0x7f1570146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:17.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.419+0000 7f15761af640 1 -- 172.21.15.112:0/970684467 shutdown_connections 2024-03-29T11:00:17.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.419+0000 7f15761af640 1 -- 172.21.15.112:0/970684467 >> 172.21.15.112:0/970684467 conn(0x7f1570066a00 msgr2=0x7f1570066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:17.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.421+0000 7f15761af640 1 -- 172.21.15.112:0/970684467 shutdown_connections 2024-03-29T11:00:17.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.421+0000 7f15761af640 1 -- 172.21.15.112:0/970684467 wait complete. 2024-03-29T11:00:17.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.422+0000 7f15761af640 10 monclient: build_initial_monmap 2024-03-29T11:00:17.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.422+0000 7f15761af640 10 monclient: monmap: 2024-03-29T11:00:17.405 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:00:17.405 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:17.405 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T11:00:17.405 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T11:00:17.405 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:17.406 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:17.406 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:00:17.406 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T11:00:17.406 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T11:00:17.406 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:17.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.422+0000 7f15761af640 1 librados: starting msgr at 2024-03-29T11:00:17.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.422+0000 7f15761af640 1 librados: starting objecter 2024-03-29T11:00:17.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.423+0000 7f15761af640 1 Processor -- start 2024-03-29T11:00:17.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.423+0000 7f15761af640 1 -- start start 2024-03-29T11:00:17.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.423+0000 7f15761af640 1 librados: setting wanted keys 2024-03-29T11:00:17.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.423+0000 7f15761af640 1 librados: calling monclient init 2024-03-29T11:00:17.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.423+0000 7f15761af640 10 monclient: init 2024-03-29T11:00:17.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.423+0000 7f15761af640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:17.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.423+0000 7f15761af640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:00:17.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.425+0000 7f15761af640 10 monclient(hunting): picked mon.noname-b con 0x7f15700b1f60 addr v1:172.21.15.118:6789/0 2024-03-29T11:00:17.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.425+0000 7f15761af640 10 monclient(hunting): picked mon.noname-a con 0x7f1570143d10 addr v1:172.21.15.112:6789/0 2024-03-29T11:00:17.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.425+0000 7f15761af640 10 monclient(hunting): picked mon.noname-c con 0x7f157006ff70 addr v1:172.21.15.112:6790/0 2024-03-29T11:00:17.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.425+0000 7f15761af640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15700703b0 con 0x7f1570143d10 2024-03-29T11:00:17.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.425+0000 7f15761af640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1570141970 con 0x7f15700b1f60 2024-03-29T11:00:17.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.425+0000 7f15761af640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1570142f10 con 0x7f157006ff70 2024-03-29T11:00:17.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.425+0000 7f15761af640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:17.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.425+0000 7f15761af640 10 monclient(hunting): authenticate will time out at 1372.730019s 2024-03-29T11:00:17.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.426+0000 7f156ffff640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f157006ff70 0x7f15701e5ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:42936/0 (socket says 172.21.15.112:42936) 2024-03-29T11:00:17.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.426+0000 7f156ffff640 1 -- 172.21.15.112:0/4060860188 learned_addr learned my addr 172.21.15.112:0/4060860188 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:00:17.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.426+0000 7f1557fff640 10 client.?.objecter ms_handle_connect 0x7f157006ff70 2024-03-29T11:00:17.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.427+0000 7f1557fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:17.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.427+0000 7f1557fff640 1 -- 172.21.15.112:0/4060860188 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078538509 0 0) 0x7f1570142f10 con 0x7f157006ff70 2024-03-29T11:00:17.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.427+0000 7f1557fff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:17.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.427+0000 7f1557fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:17.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.427+0000 7f1557fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:17.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.427+0000 7f1557fff640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:00:17.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.427+0000 7f1557fff640 1 -- 172.21.15.112:0/4060860188 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f155c004eb0 con 0x7f157006ff70 2024-03-29T11:00:17.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.427+0000 7f1557fff640 1 -- 172.21.15.112:0/4060860188 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 81870981 0 0) 0x7f155c004eb0 con 0x7f157006ff70 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.427+0000 7f1557fff640 1 -- 172.21.15.112:0/4060860188 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1570142f10 con 0x7f157006ff70 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.427+0000 7f1557fff640 1 -- 172.21.15.112:0/4060860188 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1568001ad0 con 0x7f157006ff70 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.427+0000 7f1557fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.427+0000 7f1557fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.427+0000 7f1557fff640 10 monclient(hunting): dump: 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:17.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.427+0000 7f1557fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:17.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.428+0000 7f1557fff640 1 -- 172.21.15.112:0/4060860188 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 51029414 0 0) 0x7f1570142f10 con 0x7f157006ff70 2024-03-29T11:00:17.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.428+0000 7f1557fff640 1 -- 172.21.15.112:0/4060860188 >> v1:172.21.15.118:6789/0 conn(0x7f15700b1f60 legacy=0x7f1570145970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:17.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.428+0000 7f1557fff640 1 -- 172.21.15.112:0/4060860188 >> v1:172.21.15.112:6789/0 conn(0x7f1570143d10 legacy=0x7f1570141090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:00:17.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.429+0000 7f1557fff640 10 monclient: _finish_hunting 0 2024-03-29T11:00:17.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.429+0000 7f1557fff640 1 monclient: found mon.c 2024-03-29T11:00:17.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.429+0000 7f1557fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:17.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.429+0000 7f1557fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:17.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.429+0000 7f1557fff640 1 -- 172.21.15.112:0/4060860188 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f157006f870 con 0x7f157006ff70 2024-03-29T11:00:17.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.429+0000 7f1557fff640 10 monclient: _finish_auth 0 2024-03-29T11:00:17.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.429+0000 7f1557fff640 10 monclient: _check_auth_tickets 2024-03-29T11:00:17.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.429+0000 7f1557fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:17.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.429+0000 7f15761af640 5 monclient: authenticate success, global_id 4589 2024-03-29T11:00:17.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.429+0000 7f15761af640 10 monclient: _renew_subs 2024-03-29T11:00:17.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.429+0000 7f15761af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:17.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.429+0000 7f15761af640 1 -- 172.21.15.112:0/4060860188 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f157006fb50 con 0x7f157006ff70 2024-03-29T11:00:17.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.429+0000 7f15761af640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:17.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.430+0000 7f1557fff640 1 -- 172.21.15.112:0/4060860188 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1568001c30 con 0x7f157006ff70 2024-03-29T11:00:17.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.430+0000 7f1557fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:17.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.430+0000 7f156dffb640 4 set_mon_vals no callback set 2024-03-29T11:00:17.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.430+0000 7f15761af640 10 monclient: _renew_subs 2024-03-29T11:00:17.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.430+0000 7f1557fff640 1 -- 172.21.15.112:0/4060860188 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1568004d80 con 0x7f157006ff70 2024-03-29T11:00:17.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.430+0000 7f15761af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:17.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.430+0000 7f15761af640 1 -- 172.21.15.112:0/4060860188 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1570070b20 con 0x7f157006ff70 2024-03-29T11:00:17.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.430+0000 7f15761af640 1 librados: init done 2024-03-29T11:00:17.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.430+0000 7f1557fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:17.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.430+0000 7f1557fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:17.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.430+0000 7f1557fff640 10 monclient: dump: 2024-03-29T11:00:17.417 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:17.417 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:17.417 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:17.417 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:17.417 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:17.417 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:17.417 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:17.417 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:17.417 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:17.417 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:17.417 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.431+0000 7f15761af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:17.417 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.431+0000 7f15761af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:17.417 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.431+0000 7f15761af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:17.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.431+0000 7f15761af640 1 -- 172.21.15.112:0/4060860188 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1570070840 con 0x7f157006ff70 2024-03-29T11:00:17.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.433+0000 7f1557fff640 1 -- 172.21.15.112:0/4060860188 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f1568004ee0 con 0x7f157006ff70 2024-03-29T11:00:17.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.433+0000 7f156f7fe640 10 monclient: get_auth_request con 0x7f155c06dc50 auth_method 0 2024-03-29T11:00:17.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.434+0000 7f1557fff640 1 -- 172.21.15.112:0/4060860188 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f1568084d60 con 0x7f157006ff70 2024-03-29T11:00:17.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.434+0000 7f1557fff640 10 client.4589.objecter ms_dispatch 0x7f1570000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:17.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.434+0000 7f1557fff640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:17.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.434+0000 7f1557fff640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:17.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.434+0000 7f1557fff640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-29T11:00:17.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.434+0000 7f1557fff640 10 client.4589.objecter ms_handle_connect 0x7f155c06dc50 2024-03-29T11:00:17.420 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.440+0000 7f1557fff640 1 -- 172.21.15.112:0/4060860188 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1570070840 con 0x7f157006ff70 2024-03-29T11:00:17.421 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.441+0000 7f1557fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:17.421 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.441+0000 7f1557fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:17.754 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.774+0000 7f15761af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:17.754 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.774+0000 7f15761af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:17.754 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.774+0000 7f15761af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:17.754 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.774+0000 7f15761af640 1 -- 172.21.15.112:0/4060860188 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1570070840 con 0x7f157006ff70 2024-03-29T11:00:17.755 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T11:00:17.755 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:17.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.775+0000 7f1557fff640 1 -- 172.21.15.112:0/4060860188 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f1570070840 con 0x7f157006ff70 2024-03-29T11:00:17.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.775+0000 7f1557fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:17.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.775+0000 7f1557fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:17.758 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.778+0000 7f15761af640 10 librados: watch_flush enter 2024-03-29T11:00:17.758 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.778+0000 7f15761af640 10 librados: watch_flush exit 2024-03-29T11:00:17.758 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.778+0000 7f15761af640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-29T11:00:17.758 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.778+0000 7f15761af640 10 client.4589.objecter successfully canceled tick 2024-03-29T11:00:17.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.780+0000 7f15761af640 1 -- 172.21.15.112:0/4060860188 >> v1:172.21.15.118:6816/3181073408 conn(0x7f155c06dc50 legacy=0x7f155c070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:17.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.780+0000 7f15761af640 10 monclient: shutdown 2024-03-29T11:00:17.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.780+0000 7f15761af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:17.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.780+0000 7f15761af640 1 -- 172.21.15.112:0/4060860188 >> v1:172.21.15.112:6790/0 conn(0x7f157006ff70 legacy=0x7f15701e5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:17.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.780+0000 7f15761af640 1 -- 172.21.15.112:0/4060860188 shutdown_connections 2024-03-29T11:00:17.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.780+0000 7f15761af640 1 -- 172.21.15.112:0/4060860188 >> 172.21.15.112:0/4060860188 conn(0x7f1570066a00 msgr2=0x7f1570066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:17.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.781+0000 7f15761af640 1 -- 172.21.15.112:0/4060860188 shutdown_connections 2024-03-29T11:00:17.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.781+0000 7f15761af640 1 -- 172.21.15.112:0/4060860188 wait complete. 2024-03-29T11:00:17.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:17.781+0000 7f15761af640 1 librados: shutdown 2024-03-29T11:00:17.778 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:24.585 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:24.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.757+0000 7f94b282a640 10 monclient: get_monmap_and_config 2024-03-29T11:00:24.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.757+0000 7f94b282a640 10 monclient: build_initial_monmap 2024-03-29T11:00:24.738 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.757+0000 7f94b282a640 10 monclient: monmap: 2024-03-29T11:00:24.738 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:00:24.738 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:24.738 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T11:00:24.758620+0000 2024-03-29T11:00:24.739 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T11:00:24.758620+0000 2024-03-29T11:00:24.739 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:24.739 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:24.739 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:00:24.739 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T11:00:24.739 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T11:00:24.739 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:24.739 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.759+0000 7f94b282a640 1 Processor -- start 2024-03-29T11:00:24.740 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.759+0000 7f94b282a640 1 -- start start 2024-03-29T11:00:24.740 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.759+0000 7f94b282a640 10 monclient: init 2024-03-29T11:00:24.740 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.759+0000 7f94b282a640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:24.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.760+0000 7f94b282a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:00:24.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.760+0000 7f94b282a640 10 monclient(hunting): picked mon.noname-c con 0x7f94a4034870 addr v1:172.21.15.118:6789/0 2024-03-29T11:00:24.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.760+0000 7f94b282a640 10 monclient(hunting): picked mon.noname-a con 0x7f94a402a660 addr v1:172.21.15.112:6789/0 2024-03-29T11:00:24.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.760+0000 7f94b282a640 10 monclient(hunting): picked mon.noname-b con 0x7f94a40327f0 addr v1:172.21.15.112:6790/0 2024-03-29T11:00:24.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.760+0000 7f94b282a640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a402c840 con 0x7f94a402a660 2024-03-29T11:00:24.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.760+0000 7f94b282a640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a402a140 con 0x7f94a4034870 2024-03-29T11:00:24.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.760+0000 7f94b282a640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a40c75d0 con 0x7f94a40327f0 2024-03-29T11:00:24.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.760+0000 7f94b282a640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:24.745 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.760+0000 7f94b282a640 10 monclient(hunting): authenticate will time out at 1380.064282s 2024-03-29T11:00:24.745 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.763+0000 7f94abfff640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f94a4034870 0x7f94a40c6ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:35631/0 (socket says 172.21.15.112:35631) 2024-03-29T11:00:24.745 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.763+0000 7f94abfff640 1 -- 172.21.15.112:0/1174096992 learned_addr learned my addr 172.21.15.112:0/1174096992 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:00:24.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.765+0000 7f94aaffd640 1 -- 172.21.15.112:0/1174096992 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4007544031 0 0) 0x7f94a40c75d0 con 0x7f94a40327f0 2024-03-29T11:00:24.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.765+0000 7f94aaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:24.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.765+0000 7f94aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:24.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.765+0000 7f94aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:24.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.765+0000 7f94aaffd640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:00:24.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.765+0000 7f94aaffd640 1 -- 172.21.15.112:0/1174096992 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f949c003520 con 0x7f94a40327f0 2024-03-29T11:00:24.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.765+0000 7f94aaffd640 1 -- 172.21.15.112:0/1174096992 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 937829830 0 0) 0x7f949c003520 con 0x7f94a40327f0 2024-03-29T11:00:24.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.765+0000 7f94aaffd640 1 -- 172.21.15.112:0/1174096992 >> v1:172.21.15.118:6789/0 conn(0x7f94a4034870 legacy=0x7f94a40c6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:24.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.765+0000 7f94aaffd640 1 -- 172.21.15.112:0/1174096992 >> v1:172.21.15.112:6789/0 conn(0x7f94a402a660 legacy=0x7f94a40320e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:24.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.765+0000 7f94aaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:00:24.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.765+0000 7f94aaffd640 1 monclient: found mon.noname-b 2024-03-29T11:00:24.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.765+0000 7f94aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:24.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.765+0000 7f94aaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T11:00:24.747 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.765+0000 7f94aaffd640 1 -- 172.21.15.112:0/1174096992 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a4033130 con 0x7f94a40327f0 2024-03-29T11:00:24.747 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.765+0000 7f94aaffd640 10 monclient: _finish_auth 0 2024-03-29T11:00:24.747 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.765+0000 7f94aaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:00:24.747 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.765+0000 7f94aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:24.747 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.765+0000 7f94aaffd640 1 -- 172.21.15.112:0/1174096992 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f94ac04f730 con 0x7f94a40327f0 2024-03-29T11:00:24.747 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.765+0000 7f94aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:24.747 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.765+0000 7f94aaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:24.747 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.765+0000 7f94aaffd640 10 monclient: dump: 2024-03-29T11:00:24.747 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:24.747 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:24.747 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:24.747 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:24.747 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:24.747 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:24.747 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:24.748 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:24.748 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:24.748 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:24.748 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.766+0000 7f94aaffd640 1 -- 172.21.15.112:0/1174096992 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94ac0008d0 con 0x7f94a40327f0 2024-03-29T11:00:24.748 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.766+0000 7f94aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:24.748 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.766+0000 7f94aaffd640 1 -- 172.21.15.112:0/1174096992 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f94ac0758b0 con 0x7f94a40327f0 2024-03-29T11:00:24.748 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.766+0000 7f94aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:24.748 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.766+0000 7f94aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:24.748 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.766+0000 7f94aaffd640 10 monclient: dump: 2024-03-29T11:00:24.748 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:24.748 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:24.748 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:24.748 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:24.748 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:24.748 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:24.749 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:24.749 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:24.749 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:24.749 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:24.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.766+0000 7f94b282a640 5 monclient: authenticate success, global_id 4592 2024-03-29T11:00:24.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.766+0000 7f94b282a640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:24.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.766+0000 7f94b282a640 4 set_mon_vals no callback set 2024-03-29T11:00:24.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.767+0000 7f94b282a640 10 monclient: shutdown 2024-03-29T11:00:24.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.767+0000 7f94b282a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:24.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.767+0000 7f94b282a640 1 -- 172.21.15.112:0/1174096992 >> v1:172.21.15.112:6790/0 conn(0x7f94a40327f0 legacy=0x7f94a40cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:24.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.767+0000 7f94b282a640 1 -- 172.21.15.112:0/1174096992 shutdown_connections 2024-03-29T11:00:24.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.767+0000 7f94b282a640 1 -- 172.21.15.112:0/1174096992 >> 172.21.15.112:0/1174096992 conn(0x7f94a4005910 msgr2=0x7f94a4005d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:24.751 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.769+0000 7f94b282a640 1 -- 172.21.15.112:0/1174096992 shutdown_connections 2024-03-29T11:00:24.751 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.769+0000 7f94b282a640 1 -- 172.21.15.112:0/1174096992 wait complete. 2024-03-29T11:00:24.751 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.770+0000 7f94b282a640 10 monclient: build_initial_monmap 2024-03-29T11:00:24.752 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.770+0000 7f94b282a640 10 monclient: monmap: 2024-03-29T11:00:24.752 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:00:24.752 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:24.752 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T11:00:24.752 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T11:00:24.752 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:24.752 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:24.752 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:00:24.752 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T11:00:24.752 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T11:00:24.752 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:24.752 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.771+0000 7f94b282a640 1 librados: starting msgr at 2024-03-29T11:00:24.752 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.771+0000 7f94b282a640 1 librados: starting objecter 2024-03-29T11:00:24.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.771+0000 7f94b282a640 1 Processor -- start 2024-03-29T11:00:24.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.771+0000 7f94b282a640 1 -- start start 2024-03-29T11:00:24.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.771+0000 7f94b282a640 1 librados: setting wanted keys 2024-03-29T11:00:24.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.771+0000 7f94b282a640 1 librados: calling monclient init 2024-03-29T11:00:24.753 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.771+0000 7f94b282a640 10 monclient: init 2024-03-29T11:00:24.754 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.772+0000 7f94b282a640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:24.754 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.772+0000 7f94b282a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:00:24.754 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.772+0000 7f94b282a640 10 monclient(hunting): picked mon.noname-b con 0x7f94a402a660 addr v1:172.21.15.118:6789/0 2024-03-29T11:00:24.754 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.772+0000 7f94b282a640 10 monclient(hunting): picked mon.noname-c con 0x7f94a40327f0 addr v1:172.21.15.112:6790/0 2024-03-29T11:00:24.754 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.772+0000 7f94b282a640 10 monclient(hunting): picked mon.noname-a con 0x7f94a4034870 addr v1:172.21.15.112:6789/0 2024-03-29T11:00:24.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.772+0000 7f94b282a640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a40bf130 con 0x7f94a4034870 2024-03-29T11:00:24.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.772+0000 7f94b282a640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a40c1b20 con 0x7f94a402a660 2024-03-29T11:00:24.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.772+0000 7f94b282a640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a40c2650 con 0x7f94a40327f0 2024-03-29T11:00:24.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.772+0000 7f94b282a640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:24.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.772+0000 7f94b282a640 10 monclient(hunting): authenticate will time out at 1380.077132s 2024-03-29T11:00:24.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.774+0000 7f94ab7fe640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f94a40327f0 0x7f94a40bf910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35972/0 (socket says 172.21.15.112:35972) 2024-03-29T11:00:24.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.774+0000 7f94ab7fe640 1 -- 172.21.15.112:0/847713623 learned_addr learned my addr 172.21.15.112:0/847713623 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:00:24.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.776+0000 7f94a8ff9640 10 client.?.objecter ms_handle_connect 0x7f94a40327f0 2024-03-29T11:00:24.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.776+0000 7f94a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:24.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.776+0000 7f94a8ff9640 10 client.?.objecter ms_handle_connect 0x7f94a402a660 2024-03-29T11:00:24.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.776+0000 7f94a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:24.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.776+0000 7f94a8ff9640 1 -- 172.21.15.112:0/847713623 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760052138 0 0) 0x7f94a40c1b20 con 0x7f94a402a660 2024-03-29T11:00:24.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.776+0000 7f94a8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:00:24.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.776+0000 7f94a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:24.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.776+0000 7f94a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:24.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.776+0000 7f94a8ff9640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:00:24.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.776+0000 7f94a8ff9640 1 -- 172.21.15.112:0/847713623 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9490004eb0 con 0x7f94a402a660 2024-03-29T11:00:24.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.777+0000 7f94a8ff9640 1 -- 172.21.15.112:0/847713623 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060229163 0 0) 0x7f94a40c2650 con 0x7f94a40327f0 2024-03-29T11:00:24.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.777+0000 7f94a8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:00:24.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.777+0000 7f94a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:24.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.777+0000 7f94a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:24.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.777+0000 7f94a8ff9640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:00:24.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.777+0000 7f94a8ff9640 1 -- 172.21.15.112:0/847713623 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94a40c1b20 con 0x7f94a40327f0 2024-03-29T11:00:24.760 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.777+0000 7f94a8ff9640 1 -- 172.21.15.112:0/847713623 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2805632125 0 0) 0x7f9490004eb0 con 0x7f94a402a660 2024-03-29T11:00:24.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.777+0000 7f94a8ff9640 1 -- 172.21.15.112:0/847713623 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94a40c2650 con 0x7f94a402a660 2024-03-29T11:00:24.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.777+0000 7f94a8ff9640 1 -- 172.21.15.112:0/847713623 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f94a00017c0 con 0x7f94a402a660 2024-03-29T11:00:24.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.777+0000 7f94a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:24.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.777+0000 7f94a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:24.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.777+0000 7f94a8ff9640 10 monclient(hunting): dump: 2024-03-29T11:00:24.761 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:24.761 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:24.761 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:24.761 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:24.761 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:24.761 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:24.761 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:24.761 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:24.761 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:24.761 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:24.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.778+0000 7f94a8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:24.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.778+0000 7f94a8ff9640 1 -- 172.21.15.112:0/847713623 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3881457556 0 0) 0x7f94a40c2650 con 0x7f94a402a660 2024-03-29T11:00:24.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.778+0000 7f94a8ff9640 1 -- 172.21.15.112:0/847713623 >> v1:172.21.15.112:6790/0 conn(0x7f94a40327f0 legacy=0x7f94a40bf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:24.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.778+0000 7f94a8ff9640 1 -- 172.21.15.112:0/847713623 >> v1:172.21.15.112:6789/0 conn(0x7f94a4034870 legacy=0x7f94a416b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:24.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.778+0000 7f94a8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:00:24.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.778+0000 7f94a8ff9640 1 monclient: found mon.b 2024-03-29T11:00:24.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.778+0000 7f94a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:24.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.778+0000 7f94a8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:00:24.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.778+0000 7f94a8ff9640 1 -- 172.21.15.112:0/847713623 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a40c1740 con 0x7f94a402a660 2024-03-29T11:00:24.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.778+0000 7f94a8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:00:24.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.778+0000 7f94a8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:00:24.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.778+0000 7f94a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:24.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.779+0000 7f94a8ff9640 1 -- 172.21.15.112:0/847713623 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94a0001c90 con 0x7f94a402a660 2024-03-29T11:00:24.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.779+0000 7f94a8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:24.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.779+0000 7f94a8ff9640 1 -- 172.21.15.112:0/847713623 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f94a0004960 con 0x7f94a402a660 2024-03-29T11:00:24.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.779+0000 7f94a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:24.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.779+0000 7f94a8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:24.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.779+0000 7f94a8ff9640 10 monclient: dump: 2024-03-29T11:00:24.763 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:24.763 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:24.763 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:24.763 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:24.763 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:24.763 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:24.763 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:24.763 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:24.763 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:24.763 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:24.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.780+0000 7f94a9ffb640 4 set_mon_vals no callback set 2024-03-29T11:00:24.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.782+0000 7f94b282a640 5 monclient: authenticate success, global_id 4516 2024-03-29T11:00:24.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.782+0000 7f94b282a640 10 monclient: _renew_subs 2024-03-29T11:00:24.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.782+0000 7f94b282a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:00:24.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.782+0000 7f94b282a640 1 -- 172.21.15.112:0/847713623 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a40c0020 con 0x7f94a402a660 2024-03-29T11:00:24.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.782+0000 7f94b282a640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:24.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.782+0000 7f94b282a640 10 monclient: _renew_subs 2024-03-29T11:00:24.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.782+0000 7f94b282a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:00:24.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.782+0000 7f94b282a640 1 -- 172.21.15.112:0/847713623 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a40c69f0 con 0x7f94a402a660 2024-03-29T11:00:24.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.782+0000 7f94b282a640 1 librados: init done 2024-03-29T11:00:24.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.783+0000 7f94b282a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:24.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.783+0000 7f94b282a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:24.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.783+0000 7f94b282a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:00:24.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.783+0000 7f94b282a640 1 -- 172.21.15.112:0/847713623 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94a40c1990 con 0x7f94a402a660 2024-03-29T11:00:24.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.786+0000 7f94a8ff9640 1 -- 172.21.15.112:0/847713623 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f94a0004ac0 con 0x7f94a402a660 2024-03-29T11:00:24.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.787+0000 7f94a8ff9640 1 -- 172.21.15.112:0/847713623 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f94a00848b0 con 0x7f94a402a660 2024-03-29T11:00:24.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.787+0000 7f94a8ff9640 10 client.4516.objecter ms_dispatch 0x7f94ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:24.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.787+0000 7f94ab7fe640 10 monclient: get_auth_request con 0x7f949006dd00 auth_method 0 2024-03-29T11:00:24.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.788+0000 7f94a8ff9640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:24.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.788+0000 7f94a8ff9640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:24.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.788+0000 7f94a8ff9640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-29T11:00:24.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.788+0000 7f94a8ff9640 10 client.4516.objecter ms_handle_connect 0x7f949006dd00 2024-03-29T11:00:24.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.793+0000 7f94a8ff9640 1 -- 172.21.15.112:0/847713623 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f94a0050190 con 0x7f94a402a660 2024-03-29T11:00:24.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.793+0000 7f94a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:24.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:24.793+0000 7f94a8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:25.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:25.119+0000 7f94b282a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:25.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:25.119+0000 7f94b282a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:25.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:25.119+0000 7f94b282a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:00:25.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:25.119+0000 7f94b282a640 1 -- 172.21.15.112:0/847713623 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94a4004cf0 con 0x7f94a402a660 2024-03-29T11:00:25.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:25.120+0000 7f94a8ff9640 1 -- 172.21.15.112:0/847713623 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f94a00520c0 con 0x7f94a402a660 2024-03-29T11:00:25.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:25.120+0000 7f94a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:25.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:25.120+0000 7f94a8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:25.100 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T11:00:25.100 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:25.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:25.124+0000 7f94b282a640 10 librados: watch_flush enter 2024-03-29T11:00:25.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:25.124+0000 7f94b282a640 10 librados: watch_flush exit 2024-03-29T11:00:25.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:25.124+0000 7f94b282a640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-29T11:00:25.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:25.124+0000 7f94b282a640 10 client.4516.objecter successfully canceled tick 2024-03-29T11:00:25.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:25.125+0000 7f94b282a640 1 -- 172.21.15.112:0/847713623 >> v1:172.21.15.118:6816/3181073408 conn(0x7f949006dd00 legacy=0x7f94900701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:25.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:25.125+0000 7f94b282a640 10 monclient: shutdown 2024-03-29T11:00:25.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:25.125+0000 7f94b282a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:25.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:25.125+0000 7f94b282a640 1 -- 172.21.15.112:0/847713623 >> v1:172.21.15.118:6789/0 conn(0x7f94a402a660 legacy=0x7f94a40c1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:25.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:25.125+0000 7f94b282a640 1 -- 172.21.15.112:0/847713623 shutdown_connections 2024-03-29T11:00:25.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:25.125+0000 7f94b282a640 1 -- 172.21.15.112:0/847713623 >> 172.21.15.112:0/847713623 conn(0x7f94a4005910 msgr2=0x7f94a40043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:25.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:25.126+0000 7f94b282a640 1 -- 172.21.15.112:0/847713623 shutdown_connections 2024-03-29T11:00:25.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:25.126+0000 7f94b282a640 1 -- 172.21.15.112:0/847713623 wait complete. 2024-03-29T11:00:25.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:25.127+0000 7f94b282a640 1 librados: shutdown 2024-03-29T11:00:25.129 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:31.936 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:32.086 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.106+0000 7f922861e640 10 monclient: get_monmap_and_config 2024-03-29T11:00:32.087 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.106+0000 7f922861e640 10 monclient: build_initial_monmap 2024-03-29T11:00:32.087 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.106+0000 7f922861e640 10 monclient: monmap: 2024-03-29T11:00:32.087 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:00:32.087 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:32.087 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T11:00:32.107851+0000 2024-03-29T11:00:32.087 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T11:00:32.107851+0000 2024-03-29T11:00:32.087 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:32.087 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:32.087 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:00:32.087 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T11:00:32.087 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T11:00:32.087 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:32.088 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.108+0000 7f922861e640 1 Processor -- start 2024-03-29T11:00:32.088 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.108+0000 7f922861e640 1 -- start start 2024-03-29T11:00:32.088 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.108+0000 7f922861e640 10 monclient: init 2024-03-29T11:00:32.088 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.109+0000 7f922861e640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:32.089 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.109+0000 7f922861e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:00:32.089 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.109+0000 7f922861e640 10 monclient(hunting): picked mon.noname-b con 0x7f92200adb60 addr v1:172.21.15.112:6790/0 2024-03-29T11:00:32.089 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.109+0000 7f922861e640 10 monclient(hunting): picked mon.noname-c con 0x7f92200afcf0 addr v1:172.21.15.118:6789/0 2024-03-29T11:00:32.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.109+0000 7f922861e640 10 monclient(hunting): picked mon.noname-a con 0x7f922014b960 addr v1:172.21.15.112:6789/0 2024-03-29T11:00:32.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.109+0000 7f922861e640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9220145700 con 0x7f922014b960 2024-03-29T11:00:32.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.109+0000 7f922861e640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f922006f550 con 0x7f92200afcf0 2024-03-29T11:00:32.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.109+0000 7f922861e640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92200729d0 con 0x7f92200adb60 2024-03-29T11:00:32.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.109+0000 7f922861e640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:32.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.109+0000 7f922861e640 10 monclient(hunting): authenticate will time out at 1387.413604s 2024-03-29T11:00:32.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.110+0000 7f92263ba640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f92200adb60 0x7f92200722c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35640/0 (socket says 172.21.15.112:35640) 2024-03-29T11:00:32.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.110+0000 7f92263ba640 1 -- 172.21.15.112:0/2160197656 learned_addr learned my addr 172.21.15.112:0/2160197656 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:00:32.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.111+0000 7f92253b8640 1 -- 172.21.15.112:0/2160197656 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419873607 0 0) 0x7f92200729d0 con 0x7f92200adb60 2024-03-29T11:00:32.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.111+0000 7f92253b8640 10 monclient(hunting): _negotiate 2024-03-29T11:00:32.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.111+0000 7f92253b8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:32.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.111+0000 7f92253b8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:32.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.111+0000 7f92253b8640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:00:32.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.111+0000 7f92253b8640 1 -- 172.21.15.112:0/2160197656 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9208003520 con 0x7f92200adb60 2024-03-29T11:00:32.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.111+0000 7f92253b8640 1 -- 172.21.15.112:0/2160197656 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1207132117 0 0) 0x7f9220145700 con 0x7f922014b960 2024-03-29T11:00:32.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.111+0000 7f92253b8640 10 monclient(hunting): _negotiate 2024-03-29T11:00:32.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.111+0000 7f92253b8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:32.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.111+0000 7f92253b8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:32.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.111+0000 7f92253b8640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:00:32.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.111+0000 7f92253b8640 1 -- 172.21.15.112:0/2160197656 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92200729d0 con 0x7f922014b960 2024-03-29T11:00:32.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.111+0000 7f92253b8640 1 -- 172.21.15.112:0/2160197656 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4279409526 0 0) 0x7f9208003520 con 0x7f92200adb60 2024-03-29T11:00:32.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.112+0000 7f92253b8640 1 -- 172.21.15.112:0/2160197656 >> v1:172.21.15.118:6789/0 conn(0x7f92200afcf0 legacy=0x7f9220145be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:00:32.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.112+0000 7f92253b8640 1 -- 172.21.15.112:0/2160197656 >> v1:172.21.15.112:6789/0 conn(0x7f922014b960 legacy=0x7f922014dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:32.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.112+0000 7f92253b8640 10 monclient: _finish_hunting 0 2024-03-29T11:00:32.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.112+0000 7f92253b8640 1 monclient: found mon.noname-b 2024-03-29T11:00:32.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.112+0000 7f92253b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:32.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.112+0000 7f92253b8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T11:00:32.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.112+0000 7f92253b8640 1 -- 172.21.15.112:0/2160197656 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92200af930 con 0x7f92200adb60 2024-03-29T11:00:32.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.112+0000 7f92253b8640 10 monclient: _finish_auth 0 2024-03-29T11:00:32.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.112+0000 7f92253b8640 10 monclient: _check_auth_tickets 2024-03-29T11:00:32.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.112+0000 7f92253b8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:32.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.112+0000 7f922861e640 5 monclient: authenticate success, global_id 4598 2024-03-29T11:00:32.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.112+0000 7f922861e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:32.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.112+0000 7f92253b8640 1 -- 172.21.15.112:0/2160197656 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9210001a10 con 0x7f92200adb60 2024-03-29T11:00:32.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.112+0000 7f92253b8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:32.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.112+0000 7f92253b8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:32.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.112+0000 7f92253b8640 10 monclient: dump: 2024-03-29T11:00:32.097 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:32.097 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:32.097 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:32.097 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:32.097 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:32.097 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:32.097 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:32.097 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:32.097 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:32.097 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:32.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.112+0000 7f922861e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:32.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.113+0000 7f92253b8640 1 -- 172.21.15.112:0/2160197656 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9210001c30 con 0x7f92200adb60 2024-03-29T11:00:32.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.113+0000 7f92253b8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:32.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.113+0000 7f922861e640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:32.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.113+0000 7f922861e640 4 set_mon_vals no callback set 2024-03-29T11:00:32.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.113+0000 7f92253b8640 1 -- 172.21.15.112:0/2160197656 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9210001fc0 con 0x7f92200adb60 2024-03-29T11:00:32.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.113+0000 7f92253b8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:32.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.113+0000 7f92253b8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:32.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.113+0000 7f92253b8640 10 monclient: dump: 2024-03-29T11:00:32.098 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:32.098 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:32.098 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:32.098 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:32.098 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:32.098 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:32.099 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:32.099 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:32.099 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:32.099 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:32.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.114+0000 7f922861e640 10 monclient: shutdown 2024-03-29T11:00:32.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.114+0000 7f922861e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:32.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.114+0000 7f922861e640 1 -- 172.21.15.112:0/2160197656 >> v1:172.21.15.112:6790/0 conn(0x7f92200adb60 legacy=0x7f92200722c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:32.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.114+0000 7f922861e640 1 -- 172.21.15.112:0/2160197656 shutdown_connections 2024-03-29T11:00:32.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.114+0000 7f922861e640 1 -- 172.21.15.112:0/2160197656 >> 172.21.15.112:0/2160197656 conn(0x7f9220067570 msgr2=0x7f92200b3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:32.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.114+0000 7f922861e640 1 -- 172.21.15.112:0/2160197656 shutdown_connections 2024-03-29T11:00:32.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.114+0000 7f922861e640 1 -- 172.21.15.112:0/2160197656 wait complete. 2024-03-29T11:00:32.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.115+0000 7f922861e640 10 monclient: build_initial_monmap 2024-03-29T11:00:32.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.115+0000 7f922861e640 10 monclient: monmap: 2024-03-29T11:00:32.101 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:00:32.101 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:32.101 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T11:00:32.101 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T11:00:32.101 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:32.101 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:32.101 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.115+0000 7f922861e640 1 librados: starting msgr at 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.115+0000 7f922861e640 1 librados: starting objecter 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.115+0000 7f922861e640 1 Processor -- start 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.116+0000 7f922861e640 1 -- start start 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.116+0000 7f922861e640 1 librados: setting wanted keys 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.116+0000 7f922861e640 1 librados: calling monclient init 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.116+0000 7f922861e640 10 monclient: init 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.116+0000 7f922861e640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.116+0000 7f922861e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.116+0000 7f922861e640 10 monclient(hunting): picked mon.noname-b con 0x7f92200adb60 addr v1:172.21.15.118:6789/0 2024-03-29T11:00:32.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.116+0000 7f922861e640 10 monclient(hunting): picked mon.noname-a con 0x7f92200afcf0 addr v1:172.21.15.112:6789/0 2024-03-29T11:00:32.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.116+0000 7f922861e640 10 monclient(hunting): picked mon.noname-c con 0x7f922014b960 addr v1:172.21.15.112:6790/0 2024-03-29T11:00:32.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.116+0000 7f922861e640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9220142260 con 0x7f92200afcf0 2024-03-29T11:00:32.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.116+0000 7f922861e640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92200ac980 con 0x7f92200adb60 2024-03-29T11:00:32.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.116+0000 7f922861e640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92200acc20 con 0x7f922014b960 2024-03-29T11:00:32.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.116+0000 7f922861e640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:32.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.116+0000 7f922861e640 10 monclient(hunting): authenticate will time out at 1387.421097s 2024-03-29T11:00:32.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.118+0000 7f9226bbb640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f922014b960 0x7f92200aae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35920/0 (socket says 172.21.15.112:35920) 2024-03-29T11:00:32.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.118+0000 7f9226bbb640 1 -- 172.21.15.112:0/1288532250 learned_addr learned my addr 172.21.15.112:0/1288532250 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:00:32.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.118+0000 7f921effd640 10 client.?.objecter ms_handle_connect 0x7f922014b960 2024-03-29T11:00:32.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.118+0000 7f921effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:32.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.119+0000 7f921effd640 1 -- 172.21.15.112:0/1288532250 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588804985 0 0) 0x7f92200acc20 con 0x7f922014b960 2024-03-29T11:00:32.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.119+0000 7f921effd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:32.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.119+0000 7f921effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:32.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.119+0000 7f921effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:32.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.119+0000 7f921effd640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:00:32.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.119+0000 7f921effd640 1 -- 172.21.15.112:0/1288532250 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f920c004eb0 con 0x7f922014b960 2024-03-29T11:00:32.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.119+0000 7f921effd640 1 -- 172.21.15.112:0/1288532250 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2270322505 0 0) 0x7f920c004eb0 con 0x7f922014b960 2024-03-29T11:00:32.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.119+0000 7f921effd640 1 -- 172.21.15.112:0/1288532250 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92200acc20 con 0x7f922014b960 2024-03-29T11:00:32.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.119+0000 7f921effd640 1 -- 172.21.15.112:0/1288532250 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9218001890 con 0x7f922014b960 2024-03-29T11:00:32.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.119+0000 7f921effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:32.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.119+0000 7f921effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:32.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.119+0000 7f921effd640 10 monclient(hunting): dump: 2024-03-29T11:00:32.104 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:32.104 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:32.104 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:32.104 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:32.104 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:32.105 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:32.105 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:32.105 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:32.105 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:32.105 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:32.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.119+0000 7f921effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:32.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.120+0000 7f921effd640 10 client.?.objecter ms_handle_connect 0x7f92200afcf0 2024-03-29T11:00:32.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.120+0000 7f921effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:32.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.120+0000 7f921effd640 1 -- 172.21.15.112:0/1288532250 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307687935 0 0) 0x7f9220142260 con 0x7f92200afcf0 2024-03-29T11:00:32.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.120+0000 7f921effd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:32.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.120+0000 7f921effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:32.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.120+0000 7f921effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:32.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.120+0000 7f921effd640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:00:32.105 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.120+0000 7f921effd640 1 -- 172.21.15.112:0/1288532250 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f920c004eb0 con 0x7f92200afcf0 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.120+0000 7f921effd640 10 client.?.objecter ms_handle_connect 0x7f92200adb60 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.120+0000 7f921effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.120+0000 7f921effd640 1 -- 172.21.15.112:0/1288532250 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3448449767 0 0) 0x7f92200acc20 con 0x7f922014b960 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.120+0000 7f921effd640 1 -- 172.21.15.112:0/1288532250 >> v1:172.21.15.118:6789/0 conn(0x7f92200adb60 legacy=0x7f92200ad140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.120+0000 7f921effd640 1 -- 172.21.15.112:0/1288532250 >> v1:172.21.15.112:6789/0 conn(0x7f92200afcf0 legacy=0x7f92201db7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.120+0000 7f921effd640 10 monclient: _finish_hunting 0 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.120+0000 7f921effd640 1 monclient: found mon.c 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.120+0000 7f921effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.120+0000 7f921effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.120+0000 7f921effd640 1 -- 172.21.15.112:0/1288532250 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92200a9f00 con 0x7f922014b960 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.120+0000 7f921effd640 10 monclient: _finish_auth 0 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.120+0000 7f921effd640 10 monclient: _check_auth_tickets 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.120+0000 7f921effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.121+0000 7f922861e640 5 monclient: authenticate success, global_id 4601 2024-03-29T11:00:32.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.121+0000 7f922861e640 10 monclient: _renew_subs 2024-03-29T11:00:32.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.121+0000 7f922861e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:32.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.121+0000 7f922861e640 1 -- 172.21.15.112:0/1288532250 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92200aa1b0 con 0x7f922014b960 2024-03-29T11:00:32.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.121+0000 7f922861e640 10 client.4601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:32.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.121+0000 7f922861e640 10 monclient: _renew_subs 2024-03-29T11:00:32.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.121+0000 7f922861e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:32.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.121+0000 7f922861e640 1 -- 172.21.15.112:0/1288532250 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92200aa730 con 0x7f922014b960 2024-03-29T11:00:32.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.121+0000 7f922861e640 1 librados: init done 2024-03-29T11:00:32.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.121+0000 7f921effd640 1 -- 172.21.15.112:0/1288532250 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92180019f0 con 0x7f922014b960 2024-03-29T11:00:32.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.121+0000 7f921effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:32.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.121+0000 7f921effd640 1 -- 172.21.15.112:0/1288532250 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9218003850 con 0x7f922014b960 2024-03-29T11:00:32.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.121+0000 7f921effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:32.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.121+0000 7f921effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:32.108 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.121+0000 7f921effd640 10 monclient: dump: 2024-03-29T11:00:32.109 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:32.109 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:32.109 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:32.109 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:32.109 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:32.109 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:32.109 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:32.109 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:32.109 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:32.109 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:32.109 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.121+0000 7f9224bb7640 4 set_mon_vals no callback set 2024-03-29T11:00:32.109 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.124+0000 7f921effd640 1 -- 172.21.15.112:0/1288532250 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f92180039b0 con 0x7f922014b960 2024-03-29T11:00:32.109 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.125+0000 7f921effd640 1 -- 172.21.15.112:0/1288532250 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f9218084a70 con 0x7f922014b960 2024-03-29T11:00:32.109 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.125+0000 7f921effd640 10 client.4601.objecter ms_dispatch 0x7f9220000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:32.109 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.125+0000 7f921effd640 3 client.4601.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:32.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.125+0000 7f921effd640 3 client.4601.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:32.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.125+0000 7f921effd640 20 client.4601.objecter dump_active .. 0 homeless 2024-03-29T11:00:32.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.125+0000 7f922861e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:32.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.125+0000 7f922861e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:32.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.125+0000 7f922861e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:32.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.125+0000 7f922861e640 1 -- 172.21.15.112:0/1288532250 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9200005080 con 0x7f922014b960 2024-03-29T11:00:32.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.125+0000 7f92263ba640 10 monclient: get_auth_request con 0x7f920c06ddc0 auth_method 0 2024-03-29T11:00:32.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.126+0000 7f921effd640 10 client.4601.objecter ms_handle_connect 0x7f920c06ddc0 2024-03-29T11:00:32.112 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.132+0000 7f921effd640 1 -- 172.21.15.112:0/1288532250 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9218050180 con 0x7f922014b960 2024-03-29T11:00:32.112 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.132+0000 7f921effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:32.112 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.132+0000 7f921effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:32.419 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.438+0000 7f922861e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:32.419 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.438+0000 7f922861e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:32.419 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.438+0000 7f922861e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:32.419 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.438+0000 7f922861e640 1 -- 172.21.15.112:0/1288532250 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9200005330 con 0x7f922014b960 2024-03-29T11:00:32.419 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.439+0000 7f921effd640 1 -- 172.21.15.112:0/1288532250 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f9218052280 con 0x7f922014b960 2024-03-29T11:00:32.419 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.439+0000 7f921effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:32.420 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.439+0000 7f921effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:32.420 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T11:00:32.420 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:32.423 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.443+0000 7f922861e640 10 librados: watch_flush enter 2024-03-29T11:00:32.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.443+0000 7f922861e640 10 librados: watch_flush exit 2024-03-29T11:00:32.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.443+0000 7f922861e640 20 client.4601.objecter shutdown clearing up homeless session... 2024-03-29T11:00:32.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.443+0000 7f922861e640 10 client.4601.objecter successfully canceled tick 2024-03-29T11:00:32.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.443+0000 7f922861e640 1 -- 172.21.15.112:0/1288532250 >> v1:172.21.15.118:6816/3181073408 conn(0x7f920c06ddc0 legacy=0x7f920c070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:32.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.443+0000 7f922861e640 10 monclient: shutdown 2024-03-29T11:00:32.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.443+0000 7f922861e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:32.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.443+0000 7f922861e640 1 -- 172.21.15.112:0/1288532250 >> v1:172.21.15.112:6790/0 conn(0x7f922014b960 legacy=0x7f92200aae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:32.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.443+0000 7f922861e640 1 -- 172.21.15.112:0/1288532250 shutdown_connections 2024-03-29T11:00:32.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.443+0000 7f922861e640 1 -- 172.21.15.112:0/1288532250 >> 172.21.15.112:0/1288532250 conn(0x7f9220067570 msgr2=0x7f92200b1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:32.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.443+0000 7f922861e640 1 -- 172.21.15.112:0/1288532250 shutdown_connections 2024-03-29T11:00:32.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.443+0000 7f922861e640 1 -- 172.21.15.112:0/1288532250 wait complete. 2024-03-29T11:00:32.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:32.444+0000 7f922861e640 1 librados: shutdown 2024-03-29T11:00:32.440 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:39.246 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:39.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.414+0000 7fb037d7a640 10 monclient: get_monmap_and_config 2024-03-29T11:00:39.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.414+0000 7fb037d7a640 10 monclient: build_initial_monmap 2024-03-29T11:00:39.394 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.414+0000 7fb037d7a640 10 monclient: monmap: 2024-03-29T11:00:39.394 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:00:39.394 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:39.394 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T11:00:39.415377+0000 2024-03-29T11:00:39.394 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T11:00:39.415377+0000 2024-03-29T11:00:39.395 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:39.395 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:39.395 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:00:39.395 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T11:00:39.395 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T11:00:39.395 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:39.395 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.415+0000 7fb037d7a640 1 Processor -- start 2024-03-29T11:00:39.395 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.415+0000 7fb037d7a640 1 -- start start 2024-03-29T11:00:39.395 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.415+0000 7fb037d7a640 10 monclient: init 2024-03-29T11:00:39.397 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.416+0000 7fb037d7a640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:39.397 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.416+0000 7fb037d7a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:00:39.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.416+0000 7fb037d7a640 10 monclient(hunting): picked mon.noname-c con 0x7fb0301422b0 addr v1:172.21.15.118:6789/0 2024-03-29T11:00:39.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.416+0000 7fb037d7a640 10 monclient(hunting): picked mon.noname-a con 0x7fb030142690 addr v1:172.21.15.112:6789/0 2024-03-29T11:00:39.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.416+0000 7fb037d7a640 10 monclient(hunting): picked mon.noname-b con 0x7fb0300b17d0 addr v1:172.21.15.112:6790/0 2024-03-29T11:00:39.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.416+0000 7fb037d7a640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb030142c80 con 0x7fb030142690 2024-03-29T11:00:39.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.416+0000 7fb037d7a640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0300aae40 con 0x7fb0301422b0 2024-03-29T11:00:39.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.416+0000 7fb037d7a640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb03013fbd0 con 0x7fb0300b17d0 2024-03-29T11:00:39.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.416+0000 7fb037d7a640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:39.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.416+0000 7fb037d7a640 10 monclient(hunting): authenticate will time out at 1394.721006s 2024-03-29T11:00:39.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.417+0000 7fb036317640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fb0300b17d0 0x7fb0300b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35654/0 (socket says 172.21.15.112:35654) 2024-03-29T11:00:39.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.417+0000 7fb036317640 1 -- 172.21.15.112:0/1341778202 learned_addr learned my addr 172.21.15.112:0/1341778202 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:00:39.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.417+0000 7fb035b16640 1 --1- 172.21.15.112:0/1341778202 >> v1:172.21.15.118:6789/0 conn(0x7fb0301422b0 0x7fb03013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:49352/0 (socket says 172.21.15.112:49352) 2024-03-29T11:00:39.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.417+0000 7fb034b14640 1 -- 172.21.15.112:0/1341778202 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 269914524 0 0) 0x7fb03013fbd0 con 0x7fb0300b17d0 2024-03-29T11:00:39.398 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.417+0000 7fb034b14640 10 monclient(hunting): _negotiate 2024-03-29T11:00:39.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.417+0000 7fb034b14640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:39.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.417+0000 7fb034b14640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:39.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.417+0000 7fb034b14640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:00:39.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.417+0000 7fb034b14640 1 -- 172.21.15.112:0/1341778202 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb01c003520 con 0x7fb0300b17d0 2024-03-29T11:00:39.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.417+0000 7fb034b14640 1 -- 172.21.15.112:0/1341778202 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1702832237 0 0) 0x7fb030142c80 con 0x7fb030142690 2024-03-29T11:00:39.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.417+0000 7fb034b14640 10 monclient(hunting): _negotiate 2024-03-29T11:00:39.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:39.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:39.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:00:39.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 1 -- 172.21.15.112:0/1341778202 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb03013fbd0 con 0x7fb030142690 2024-03-29T11:00:39.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 1 -- 172.21.15.112:0/1341778202 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765326331 0 0) 0x7fb0300aae40 con 0x7fb0301422b0 2024-03-29T11:00:39.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 10 monclient(hunting): _negotiate 2024-03-29T11:00:39.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:39.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:39.399 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:00:39.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 1 -- 172.21.15.112:0/1341778202 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb030142c80 con 0x7fb0301422b0 2024-03-29T11:00:39.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 1 -- 172.21.15.112:0/1341778202 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2442158129 0 0) 0x7fb03013fbd0 con 0x7fb030142690 2024-03-29T11:00:39.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 1 -- 172.21.15.112:0/1341778202 >> v1:172.21.15.112:6790/0 conn(0x7fb0300b17d0 legacy=0x7fb0300b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:39.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 1 -- 172.21.15.112:0/1341778202 >> v1:172.21.15.118:6789/0 conn(0x7fb0301422b0 legacy=0x7fb03013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:39.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 10 monclient: _finish_hunting 0 2024-03-29T11:00:39.405 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 1 monclient: found mon.noname-a 2024-03-29T11:00:39.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:39.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T11:00:39.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 1 -- 172.21.15.112:0/1341778202 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb030151940 con 0x7fb030142690 2024-03-29T11:00:39.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 10 monclient: _finish_auth 0 2024-03-29T11:00:39.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 10 monclient: _check_auth_tickets 2024-03-29T11:00:39.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:39.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb037d7a640 5 monclient: authenticate success, global_id 4545 2024-03-29T11:00:39.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb037d7a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:39.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 1 -- 172.21.15.112:0/1341778202 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb0200015a0 con 0x7fb030142690 2024-03-29T11:00:39.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:39.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:00:39.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 10 monclient: dump: 2024-03-29T11:00:39.406 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:39.406 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:39.406 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:39.407 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:39.407 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:39.407 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:39.407 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:39.407 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:39.407 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:39.407 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:39.407 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb037d7a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:39.407 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 1 -- 172.21.15.112:0/1341778202 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0200017c0 con 0x7fb030142690 2024-03-29T11:00:39.407 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb034b14640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:39.407 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb037d7a640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:39.407 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.418+0000 7fb037d7a640 4 set_mon_vals no callback set 2024-03-29T11:00:39.407 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.419+0000 7fb034b14640 1 -- 172.21.15.112:0/1341778202 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb020001ad0 con 0x7fb030142690 2024-03-29T11:00:39.407 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.419+0000 7fb034b14640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:39.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.419+0000 7fb034b14640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:39.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.419+0000 7fb034b14640 10 monclient: dump: 2024-03-29T11:00:39.408 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:39.408 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:39.408 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:39.408 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:39.408 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:39.408 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:39.408 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:39.408 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:39.408 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:39.408 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:39.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.419+0000 7fb037d7a640 10 monclient: shutdown 2024-03-29T11:00:39.408 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.419+0000 7fb037d7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:39.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.419+0000 7fb037d7a640 1 -- 172.21.15.112:0/1341778202 >> v1:172.21.15.112:6789/0 conn(0x7fb030142690 legacy=0x7fb0300b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:39.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.419+0000 7fb037d7a640 1 -- 172.21.15.112:0/1341778202 shutdown_connections 2024-03-29T11:00:39.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.419+0000 7fb037d7a640 1 -- 172.21.15.112:0/1341778202 >> 172.21.15.112:0/1341778202 conn(0x7fb030066a00 msgr2=0x7fb030066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:39.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.419+0000 7fb037d7a640 1 -- 172.21.15.112:0/1341778202 shutdown_connections 2024-03-29T11:00:39.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.419+0000 7fb037d7a640 1 -- 172.21.15.112:0/1341778202 wait complete. 2024-03-29T11:00:39.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.420+0000 7fb037d7a640 10 monclient: build_initial_monmap 2024-03-29T11:00:39.409 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.420+0000 7fb037d7a640 10 monclient: monmap: 2024-03-29T11:00:39.409 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:00:39.409 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:39.409 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T11:00:39.409 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T11:00:39.409 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:39.409 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:39.409 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:00:39.409 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T11:00:39.409 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T11:00:39.410 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:39.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.421+0000 7fb037d7a640 1 librados: starting msgr at 2024-03-29T11:00:39.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.421+0000 7fb037d7a640 1 librados: starting objecter 2024-03-29T11:00:39.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.421+0000 7fb037d7a640 1 Processor -- start 2024-03-29T11:00:39.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.421+0000 7fb037d7a640 1 -- start start 2024-03-29T11:00:39.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.421+0000 7fb037d7a640 1 librados: setting wanted keys 2024-03-29T11:00:39.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.421+0000 7fb037d7a640 1 librados: calling monclient init 2024-03-29T11:00:39.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.421+0000 7fb037d7a640 10 monclient: init 2024-03-29T11:00:39.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.422+0000 7fb037d7a640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:39.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.422+0000 7fb037d7a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:00:39.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.422+0000 7fb037d7a640 10 monclient(hunting): picked mon.noname-b con 0x7fb0300b17d0 addr v1:172.21.15.118:6789/0 2024-03-29T11:00:39.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.422+0000 7fb037d7a640 10 monclient(hunting): picked mon.noname-a con 0x7fb0301422b0 addr v1:172.21.15.112:6789/0 2024-03-29T11:00:39.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.422+0000 7fb035315640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fb0301422b0 0x7fb030132000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:55602/0 (socket says 172.21.15.112:55602) 2024-03-29T11:00:39.410 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.422+0000 7fb037d7a640 10 monclient(hunting): picked mon.noname-c con 0x7fb030142690 addr v1:172.21.15.112:6790/0 2024-03-29T11:00:39.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.422+0000 7fb035315640 1 -- 172.21.15.112:0/1790728807 learned_addr learned my addr 172.21.15.112:0/1790728807 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:00:39.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.422+0000 7fb037d7a640 1 -- 172.21.15.112:0/1790728807 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0301dfc70 con 0x7fb0301422b0 2024-03-29T11:00:39.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.422+0000 7fb037d7a640 1 -- 172.21.15.112:0/1790728807 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb03006ed00 con 0x7fb0300b17d0 2024-03-29T11:00:39.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.422+0000 7fb037d7a640 1 -- 172.21.15.112:0/1790728807 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0301df040 con 0x7fb030142690 2024-03-29T11:00:39.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.422+0000 7fb037d7a640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:39.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.422+0000 7fb037d7a640 10 monclient(hunting): authenticate will time out at 1394.726806s 2024-03-29T11:00:39.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.422+0000 7fb0267fc640 10 client.?.objecter ms_handle_connect 0x7fb0301422b0 2024-03-29T11:00:39.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.422+0000 7fb0267fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:39.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.422+0000 7fb0267fc640 10 client.?.objecter ms_handle_connect 0x7fb030142690 2024-03-29T11:00:39.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.422+0000 7fb0267fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:39.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.422+0000 7fb0267fc640 1 -- 172.21.15.112:0/1790728807 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469052187 0 0) 0x7fb0301df040 con 0x7fb030142690 2024-03-29T11:00:39.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.422+0000 7fb0267fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:39.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.422+0000 7fb0267fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:39.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.422+0000 7fb0267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:39.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.422+0000 7fb0267fc640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:00:39.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.422+0000 7fb0267fc640 1 -- 172.21.15.112:0/1790728807 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb014004eb0 con 0x7fb030142690 2024-03-29T11:00:39.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.423+0000 7fb0267fc640 10 client.?.objecter ms_handle_connect 0x7fb0300b17d0 2024-03-29T11:00:39.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.423+0000 7fb0267fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:39.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.423+0000 7fb0267fc640 1 -- 172.21.15.112:0/1790728807 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1441268650 0 0) 0x7fb014004eb0 con 0x7fb030142690 2024-03-29T11:00:39.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.423+0000 7fb0267fc640 1 -- 172.21.15.112:0/1790728807 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0301df040 con 0x7fb030142690 2024-03-29T11:00:39.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.423+0000 7fb0267fc640 1 -- 172.21.15.112:0/1790728807 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb02c001150 con 0x7fb030142690 2024-03-29T11:00:39.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.423+0000 7fb0267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:39.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.423+0000 7fb0267fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:39.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.423+0000 7fb0267fc640 10 monclient(hunting): dump: 2024-03-29T11:00:39.412 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:39.412 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:39.412 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:39.412 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:39.413 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:39.413 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:39.413 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:39.413 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:39.413 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:39.413 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:39.413 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.423+0000 7fb0267fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:39.413 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.423+0000 7fb0267fc640 1 -- 172.21.15.112:0/1790728807 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1812960179 0 0) 0x7fb03006ed00 con 0x7fb0300b17d0 2024-03-29T11:00:39.413 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.423+0000 7fb0267fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:39.413 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.423+0000 7fb0267fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:39.413 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.423+0000 7fb0267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:39.413 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.423+0000 7fb0267fc640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:00:39.413 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.423+0000 7fb0267fc640 1 -- 172.21.15.112:0/1790728807 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb014004eb0 con 0x7fb0300b17d0 2024-03-29T11:00:39.413 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.423+0000 7fb0267fc640 1 -- 172.21.15.112:0/1790728807 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124484937 0 0) 0x7fb0301dfc70 con 0x7fb0301422b0 2024-03-29T11:00:39.413 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.423+0000 7fb0267fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:39.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.423+0000 7fb0267fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:39.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.423+0000 7fb0267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:39.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.423+0000 7fb0267fc640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:00:39.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.423+0000 7fb0267fc640 1 -- 172.21.15.112:0/1790728807 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb014005860 con 0x7fb0301422b0 2024-03-29T11:00:39.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb0267fc640 1 -- 172.21.15.112:0/1790728807 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3099717161 0 0) 0x7fb014004eb0 con 0x7fb0300b17d0 2024-03-29T11:00:39.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb0267fc640 1 -- 172.21.15.112:0/1790728807 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0301dfc70 con 0x7fb0300b17d0 2024-03-29T11:00:39.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb0267fc640 1 -- 172.21.15.112:0/1790728807 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb028001bf0 con 0x7fb0300b17d0 2024-03-29T11:00:39.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb0267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:39.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb0267fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:39.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb0267fc640 10 monclient(hunting): dump: 2024-03-29T11:00:39.414 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:39.415 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:39.415 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:39.415 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:39.415 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:39.415 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:39.415 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:39.415 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:39.415 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:39.415 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:39.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb0267fc640 1 -- 172.21.15.112:0/1790728807 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3272429391 0 0) 0x7fb014005860 con 0x7fb0301422b0 2024-03-29T11:00:39.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb0267fc640 1 -- 172.21.15.112:0/1790728807 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb014004eb0 con 0x7fb0301422b0 2024-03-29T11:00:39.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb0267fc640 1 -- 172.21.15.112:0/1790728807 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2996279124 0 0) 0x7fb0301df040 con 0x7fb030142690 2024-03-29T11:00:39.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb0267fc640 1 -- 172.21.15.112:0/1790728807 >> v1:172.21.15.118:6789/0 conn(0x7fb0300b17d0 legacy=0x7fb0301df560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:39.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb0267fc640 1 -- 172.21.15.112:0/1790728807 >> v1:172.21.15.112:6789/0 conn(0x7fb0301422b0 legacy=0x7fb030132000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:39.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb0267fc640 10 monclient: _finish_hunting 0 2024-03-29T11:00:39.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb0267fc640 1 monclient: found mon.c 2024-03-29T11:00:39.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb0267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:39.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb0267fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:39.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb0267fc640 1 -- 172.21.15.112:0/1790728807 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb03006de90 con 0x7fb030142690 2024-03-29T11:00:39.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb0267fc640 10 monclient: _finish_auth 0 2024-03-29T11:00:39.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb0267fc640 10 monclient: _check_auth_tickets 2024-03-29T11:00:39.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb0267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:39.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb037d7a640 5 monclient: authenticate success, global_id 4607 2024-03-29T11:00:39.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb037d7a640 10 monclient: _renew_subs 2024-03-29T11:00:39.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb037d7a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:39.417 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb037d7a640 1 -- 172.21.15.112:0/1790728807 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb030071450 con 0x7fb030142690 2024-03-29T11:00:39.417 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb037d7a640 10 client.4607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:39.417 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb037d7a640 10 monclient: _renew_subs 2024-03-29T11:00:39.417 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb037d7a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:39.417 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb037d7a640 1 -- 172.21.15.112:0/1790728807 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0300717f0 con 0x7fb030142690 2024-03-29T11:00:39.417 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.424+0000 7fb037d7a640 1 librados: init done 2024-03-29T11:00:39.417 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.425+0000 7fb037d7a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:39.417 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.425+0000 7fb037d7a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:39.417 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.425+0000 7fb037d7a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:39.417 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.425+0000 7fb037d7a640 1 -- 172.21.15.112:0/1790728807 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0301df1d0 con 0x7fb030142690 2024-03-29T11:00:39.417 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.425+0000 7fb0267fc640 1 -- 172.21.15.112:0/1790728807 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb02c001640 con 0x7fb030142690 2024-03-29T11:00:39.417 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.425+0000 7fb0267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:39.419 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.425+0000 7fb027fff640 4 set_mon_vals no callback set 2024-03-29T11:00:39.419 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.426+0000 7fb0267fc640 1 -- 172.21.15.112:0/1790728807 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb02c004980 con 0x7fb030142690 2024-03-29T11:00:39.419 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.426+0000 7fb0267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:39.419 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.426+0000 7fb0267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:39.419 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.426+0000 7fb0267fc640 10 monclient: dump: 2024-03-29T11:00:39.420 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:39.420 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:39.420 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:39.420 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:39.420 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:39.420 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:39.420 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:39.420 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:39.420 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:39.420 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:39.420 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.427+0000 7fb0267fc640 1 -- 172.21.15.112:0/1790728807 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fb02c01a950 con 0x7fb030142690 2024-03-29T11:00:39.420 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.428+0000 7fb035b16640 10 monclient: get_auth_request con 0x7fb01406e2e0 auth_method 0 2024-03-29T11:00:39.420 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.428+0000 7fb0267fc640 1 -- 172.21.15.112:0/1790728807 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7fb02c0848d0 con 0x7fb030142690 2024-03-29T11:00:39.420 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.428+0000 7fb0267fc640 10 client.4607.objecter ms_dispatch 0x7fb030000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:39.420 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.429+0000 7fb0267fc640 3 client.4607.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:39.421 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.429+0000 7fb0267fc640 3 client.4607.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:39.421 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.429+0000 7fb0267fc640 20 client.4607.objecter dump_active .. 0 homeless 2024-03-29T11:00:39.421 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.429+0000 7fb0267fc640 10 client.4607.objecter ms_handle_connect 0x7fb01406e2e0 2024-03-29T11:00:39.421 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.437+0000 7fb0267fc640 1 -- 172.21.15.112:0/1790728807 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb02c04ffe0 con 0x7fb030142690 2024-03-29T11:00:39.421 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.438+0000 7fb0267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:39.421 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.438+0000 7fb0267fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:39.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.787+0000 7fb037d7a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:39.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.787+0000 7fb037d7a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:39.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.787+0000 7fb037d7a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:39.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.787+0000 7fb037d7a640 1 -- 172.21.15.112:0/1790728807 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0301dee30 con 0x7fb030142690 2024-03-29T11:00:39.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.788+0000 7fb0267fc640 1 -- 172.21.15.112:0/1790728807 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7fb02c0520e0 con 0x7fb030142690 2024-03-29T11:00:39.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.788+0000 7fb0267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:39.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.788+0000 7fb0267fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:39.768 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T11:00:39.768 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:39.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.791+0000 7fb037d7a640 10 librados: watch_flush enter 2024-03-29T11:00:39.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.791+0000 7fb037d7a640 10 librados: watch_flush exit 2024-03-29T11:00:39.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.791+0000 7fb037d7a640 20 client.4607.objecter shutdown clearing up homeless session... 2024-03-29T11:00:39.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.791+0000 7fb037d7a640 10 client.4607.objecter successfully canceled tick 2024-03-29T11:00:39.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.791+0000 7fb037d7a640 1 -- 172.21.15.112:0/1790728807 >> v1:172.21.15.118:6816/3181073408 conn(0x7fb01406e2e0 legacy=0x7fb0140707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:39.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.791+0000 7fb037d7a640 10 monclient: shutdown 2024-03-29T11:00:39.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.791+0000 7fb037d7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:39.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.791+0000 7fb037d7a640 1 -- 172.21.15.112:0/1790728807 >> v1:172.21.15.112:6790/0 conn(0x7fb030142690 legacy=0x7fb0301e70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:39.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.791+0000 7fb037d7a640 1 -- 172.21.15.112:0/1790728807 shutdown_connections 2024-03-29T11:00:39.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.791+0000 7fb037d7a640 1 -- 172.21.15.112:0/1790728807 >> 172.21.15.112:0/1790728807 conn(0x7fb030066a00 msgr2=0x7fb03013c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:39.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.791+0000 7fb037d7a640 1 -- 172.21.15.112:0/1790728807 shutdown_connections 2024-03-29T11:00:39.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.792+0000 7fb037d7a640 1 -- 172.21.15.112:0/1790728807 wait complete. 2024-03-29T11:00:39.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:39.792+0000 7fb037d7a640 1 librados: shutdown 2024-03-29T11:00:39.787 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:46.593 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:46.757 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.774+0000 7fcb35944640 10 monclient: get_monmap_and_config 2024-03-29T11:00:46.757 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.774+0000 7fcb35944640 10 monclient: build_initial_monmap 2024-03-29T11:00:46.757 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.775+0000 7fcb35944640 10 monclient: monmap: 2024-03-29T11:00:46.757 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:00:46.757 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:46.757 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T11:00:46.775935+0000 2024-03-29T11:00:46.757 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T11:00:46.775935+0000 2024-03-29T11:00:46.757 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:46.757 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:46.757 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:00:46.757 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T11:00:46.757 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T11:00:46.758 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:46.758 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.776+0000 7fcb35944640 1 Processor -- start 2024-03-29T11:00:46.758 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.776+0000 7fcb35944640 1 -- start start 2024-03-29T11:00:46.758 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.776+0000 7fcb35944640 10 monclient: init 2024-03-29T11:00:46.758 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.777+0000 7fcb35944640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:46.758 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.777+0000 7fcb35944640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:00:46.758 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.777+0000 7fcb35944640 10 monclient(hunting): picked mon.noname-b con 0x7fcb300af6c0 addr v1:172.21.15.112:6790/0 2024-03-29T11:00:46.758 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.777+0000 7fcb35944640 10 monclient(hunting): picked mon.noname-a con 0x7fcb3014f6b0 addr v1:172.21.15.112:6789/0 2024-03-29T11:00:46.758 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.777+0000 7fcb35944640 10 monclient(hunting): picked mon.noname-c con 0x7fcb30152980 addr v1:172.21.15.118:6789/0 2024-03-29T11:00:46.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.777+0000 7fcb35944640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb300a94e0 con 0x7fcb3014f6b0 2024-03-29T11:00:46.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.777+0000 7fcb35944640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb300afb60 con 0x7fcb30152980 2024-03-29T11:00:46.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.777+0000 7fcb35944640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb300aab80 con 0x7fcb300af6c0 2024-03-29T11:00:46.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.777+0000 7fcb35944640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:46.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.777+0000 7fcb35944640 10 monclient(hunting): authenticate will time out at 1402.081700s 2024-03-29T11:00:46.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.781+0000 7fcb2f7fe640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fcb30152980 0x7fcb30152e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:41303/0 (socket says 172.21.15.112:41303) 2024-03-29T11:00:46.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.781+0000 7fcb2f7fe640 1 -- 172.21.15.112:0/168816856 learned_addr learned my addr 172.21.15.112:0/168816856 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:00:46.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.782+0000 7fcb2dffb640 1 -- 172.21.15.112:0/168816856 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634822126 0 0) 0x7fcb300afb60 con 0x7fcb30152980 2024-03-29T11:00:46.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.782+0000 7fcb2dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:46.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.782+0000 7fcb2dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:46.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.782+0000 7fcb2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:46.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.782+0000 7fcb2dffb640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:00:46.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.782+0000 7fcb2dffb640 1 -- 172.21.15.112:0/168816856 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb18003520 con 0x7fcb30152980 2024-03-29T11:00:46.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.782+0000 7fcb2dffb640 1 -- 172.21.15.112:0/168816856 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 119879178 0 0) 0x7fcb18003520 con 0x7fcb30152980 2024-03-29T11:00:46.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.782+0000 7fcb2dffb640 1 -- 172.21.15.112:0/168816856 >> v1:172.21.15.112:6790/0 conn(0x7fcb300af6c0 legacy=0x7fcb300ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:46.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.782+0000 7fcb2dffb640 1 -- 172.21.15.112:0/168816856 >> v1:172.21.15.112:6789/0 conn(0x7fcb3014f6b0 legacy=0x7fcb3014fae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:00:46.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.783+0000 7fcb2dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:00:46.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.783+0000 7fcb2dffb640 1 monclient: found mon.noname-c 2024-03-29T11:00:46.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.783+0000 7fcb2dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:46.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.783+0000 7fcb2dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T11:00:46.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.783+0000 7fcb2dffb640 1 -- 172.21.15.112:0/168816856 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb300aa680 con 0x7fcb30152980 2024-03-29T11:00:46.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.783+0000 7fcb2dffb640 10 monclient: _finish_auth 0 2024-03-29T11:00:46.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.783+0000 7fcb2dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:00:46.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.783+0000 7fcb2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:46.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.783+0000 7fcb2dffb640 1 -- 172.21.15.112:0/168816856 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fcb28001a10 con 0x7fcb30152980 2024-03-29T11:00:46.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.783+0000 7fcb2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:46.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.783+0000 7fcb2dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:46.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.783+0000 7fcb2dffb640 10 monclient: dump: 2024-03-29T11:00:46.764 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:46.764 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:46.764 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:46.765 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:46.765 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:46.765 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:46.765 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:46.765 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:46.765 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:46.765 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:46.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.783+0000 7fcb35944640 5 monclient: authenticate success, global_id 4528 2024-03-29T11:00:46.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.783+0000 7fcb35944640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:46.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.784+0000 7fcb2dffb640 1 -- 172.21.15.112:0/168816856 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb28001be0 con 0x7fcb30152980 2024-03-29T11:00:46.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.784+0000 7fcb2dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:46.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.784+0000 7fcb2dffb640 1 -- 172.21.15.112:0/168816856 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fcb28002030 con 0x7fcb30152980 2024-03-29T11:00:46.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.784+0000 7fcb2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:46.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.784+0000 7fcb2dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:46.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.784+0000 7fcb2dffb640 10 monclient: dump: 2024-03-29T11:00:46.766 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:46.766 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:46.766 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:46.766 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:46.767 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:46.767 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:46.767 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:46.767 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:46.767 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:46.767 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:46.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.784+0000 7fcb35944640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:46.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.784+0000 7fcb35944640 4 set_mon_vals no callback set 2024-03-29T11:00:46.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.785+0000 7fcb35944640 10 monclient: shutdown 2024-03-29T11:00:46.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.785+0000 7fcb35944640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:46.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.785+0000 7fcb35944640 1 -- 172.21.15.112:0/168816856 >> v1:172.21.15.118:6789/0 conn(0x7fcb30152980 legacy=0x7fcb30152e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:46.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.785+0000 7fcb35944640 1 -- 172.21.15.112:0/168816856 shutdown_connections 2024-03-29T11:00:46.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.785+0000 7fcb35944640 1 -- 172.21.15.112:0/168816856 >> 172.21.15.112:0/168816856 conn(0x7fcb300b35e0 msgr2=0x7fcb300b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:46.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.787+0000 7fcb35944640 1 -- 172.21.15.112:0/168816856 shutdown_connections 2024-03-29T11:00:46.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.787+0000 7fcb35944640 1 -- 172.21.15.112:0/168816856 wait complete. 2024-03-29T11:00:46.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.788+0000 7fcb35944640 10 monclient: build_initial_monmap 2024-03-29T11:00:46.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.788+0000 7fcb35944640 10 monclient: monmap: 2024-03-29T11:00:46.769 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:00:46.770 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:46.770 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T11:00:46.770 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T11:00:46.770 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:46.770 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:46.770 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:00:46.770 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T11:00:46.770 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T11:00:46.770 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:46.770 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.789+0000 7fcb35944640 1 librados: starting msgr at 2024-03-29T11:00:46.770 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.789+0000 7fcb35944640 1 librados: starting objecter 2024-03-29T11:00:46.770 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.789+0000 7fcb35944640 1 Processor -- start 2024-03-29T11:00:46.770 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.789+0000 7fcb35944640 1 -- start start 2024-03-29T11:00:46.770 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.789+0000 7fcb35944640 1 librados: setting wanted keys 2024-03-29T11:00:46.771 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.789+0000 7fcb35944640 1 librados: calling monclient init 2024-03-29T11:00:46.771 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.789+0000 7fcb35944640 10 monclient: init 2024-03-29T11:00:46.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.790+0000 7fcb35944640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:46.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.790+0000 7fcb35944640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:00:46.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.790+0000 7fcb35944640 10 monclient(hunting): picked mon.noname-b con 0x7fcb300af6c0 addr v1:172.21.15.118:6789/0 2024-03-29T11:00:46.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.790+0000 7fcb35944640 10 monclient(hunting): picked mon.noname-c con 0x7fcb3014f6b0 addr v1:172.21.15.112:6790/0 2024-03-29T11:00:46.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.790+0000 7fcb35944640 10 monclient(hunting): picked mon.noname-a con 0x7fcb30152980 addr v1:172.21.15.112:6789/0 2024-03-29T11:00:46.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.790+0000 7fcb35944640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb3007dc30 con 0x7fcb30152980 2024-03-29T11:00:46.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.790+0000 7fcb35944640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb3014eed0 con 0x7fcb300af6c0 2024-03-29T11:00:46.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.790+0000 7fcb35944640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb3014f060 con 0x7fcb3014f6b0 2024-03-29T11:00:46.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.790+0000 7fcb35944640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:46.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.790+0000 7fcb35944640 10 monclient(hunting): authenticate will time out at 1402.094761s 2024-03-29T11:00:46.777 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.794+0000 7fcb2e7fc640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fcb3014f6b0 0x7fcb301e3c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:50664/0 (socket says 172.21.15.112:50664) 2024-03-29T11:00:46.777 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.794+0000 7fcb2e7fc640 1 -- 172.21.15.112:0/625686584 learned_addr learned my addr 172.21.15.112:0/625686584 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:00:46.777 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.796+0000 7fcb17fff640 10 client.?.objecter ms_handle_connect 0x7fcb300af6c0 2024-03-29T11:00:46.777 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.796+0000 7fcb17fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:46.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.796+0000 7fcb17fff640 10 client.?.objecter ms_handle_connect 0x7fcb3014f6b0 2024-03-29T11:00:46.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.796+0000 7fcb17fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:46.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.796+0000 7fcb17fff640 1 -- 172.21.15.112:0/625686584 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3609954049 0 0) 0x7fcb3014eed0 con 0x7fcb300af6c0 2024-03-29T11:00:46.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.796+0000 7fcb17fff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:46.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.796+0000 7fcb17fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:46.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.796+0000 7fcb17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:46.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.796+0000 7fcb17fff640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:00:46.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.796+0000 7fcb17fff640 1 -- 172.21.15.112:0/625686584 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb1c004eb0 con 0x7fcb300af6c0 2024-03-29T11:00:46.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.796+0000 7fcb17fff640 1 -- 172.21.15.112:0/625686584 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1768371972 0 0) 0x7fcb3014f060 con 0x7fcb3014f6b0 2024-03-29T11:00:46.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.796+0000 7fcb17fff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:46.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.796+0000 7fcb17fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:46.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.796+0000 7fcb17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:46.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.796+0000 7fcb17fff640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:00:46.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.796+0000 7fcb17fff640 1 -- 172.21.15.112:0/625686584 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb3014eed0 con 0x7fcb3014f6b0 2024-03-29T11:00:46.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.797+0000 7fcb17fff640 10 client.?.objecter ms_handle_connect 0x7fcb30152980 2024-03-29T11:00:46.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.797+0000 7fcb17fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:46.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.797+0000 7fcb17fff640 1 -- 172.21.15.112:0/625686584 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500536127 0 0) 0x7fcb3007dc30 con 0x7fcb30152980 2024-03-29T11:00:46.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.797+0000 7fcb17fff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:46.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.797+0000 7fcb17fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:46.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.797+0000 7fcb17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:46.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.797+0000 7fcb17fff640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:00:46.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.797+0000 7fcb17fff640 1 -- 172.21.15.112:0/625686584 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb3014f060 con 0x7fcb30152980 2024-03-29T11:00:46.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.797+0000 7fcb17fff640 1 -- 172.21.15.112:0/625686584 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3834620158 0 0) 0x7fcb1c004eb0 con 0x7fcb300af6c0 2024-03-29T11:00:46.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.797+0000 7fcb17fff640 1 -- 172.21.15.112:0/625686584 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb3007dc30 con 0x7fcb300af6c0 2024-03-29T11:00:46.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.797+0000 7fcb17fff640 1 -- 172.21.15.112:0/625686584 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fcb20001930 con 0x7fcb300af6c0 2024-03-29T11:00:46.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.797+0000 7fcb17fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:46.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.797+0000 7fcb17fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:46.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.797+0000 7fcb17fff640 10 monclient(hunting): dump: 2024-03-29T11:00:46.780 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:46.780 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:46.780 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:46.780 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:46.780 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:46.780 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:46.780 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:46.780 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:46.780 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:46.780 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:46.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.797+0000 7fcb17fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:46.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.797+0000 7fcb17fff640 1 -- 172.21.15.112:0/625686584 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1574073923 0 0) 0x7fcb3014eed0 con 0x7fcb3014f6b0 2024-03-29T11:00:46.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.797+0000 7fcb17fff640 1 -- 172.21.15.112:0/625686584 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb1c004eb0 con 0x7fcb3014f6b0 2024-03-29T11:00:46.780 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.797+0000 7fcb17fff640 1 -- 172.21.15.112:0/625686584 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fcb240017d0 con 0x7fcb3014f6b0 2024-03-29T11:00:46.781 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.797+0000 7fcb17fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:46.781 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.797+0000 7fcb17fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:46.781 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.797+0000 7fcb17fff640 10 monclient(hunting): dump: 2024-03-29T11:00:46.781 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:46.781 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:46.781 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:46.781 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:46.781 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:46.781 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:46.781 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:46.781 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:46.781 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:46.781 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:46.782 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.801+0000 7fcb17fff640 1 -- 172.21.15.112:0/625686584 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 543501624 0 0) 0x7fcb1c004eb0 con 0x7fcb3014f6b0 2024-03-29T11:00:46.782 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.801+0000 7fcb17fff640 1 -- 172.21.15.112:0/625686584 >> v1:172.21.15.118:6789/0 conn(0x7fcb300af6c0 legacy=0x7fcb3014e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:46.782 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.801+0000 7fcb17fff640 1 -- 172.21.15.112:0/625686584 >> v1:172.21.15.112:6789/0 conn(0x7fcb30152980 legacy=0x7fcb301e4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:46.782 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.801+0000 7fcb17fff640 10 monclient: _finish_hunting 0 2024-03-29T11:00:46.782 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.801+0000 7fcb17fff640 1 monclient: found mon.c 2024-03-29T11:00:46.782 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.801+0000 7fcb17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:46.782 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.801+0000 7fcb17fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:46.782 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.801+0000 7fcb17fff640 1 -- 172.21.15.112:0/625686584 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3007cfc0 con 0x7fcb3014f6b0 2024-03-29T11:00:46.782 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.801+0000 7fcb17fff640 10 monclient: _finish_auth 0 2024-03-29T11:00:46.782 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.801+0000 7fcb17fff640 10 monclient: _check_auth_tickets 2024-03-29T11:00:46.782 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.801+0000 7fcb17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:46.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.801+0000 7fcb35944640 5 monclient: authenticate success, global_id 4613 2024-03-29T11:00:46.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.801+0000 7fcb35944640 10 monclient: _renew_subs 2024-03-29T11:00:46.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.801+0000 7fcb35944640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:46.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.801+0000 7fcb35944640 1 -- 172.21.15.112:0/625686584 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb3007d1b0 con 0x7fcb3014f6b0 2024-03-29T11:00:46.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.801+0000 7fcb35944640 10 client.4613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:46.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.801+0000 7fcb35944640 10 monclient: _renew_subs 2024-03-29T11:00:46.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.801+0000 7fcb35944640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:46.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.801+0000 7fcb35944640 1 -- 172.21.15.112:0/625686584 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb30071e90 con 0x7fcb3014f6b0 2024-03-29T11:00:46.783 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.801+0000 7fcb35944640 1 librados: init done 2024-03-29T11:00:46.784 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.801+0000 7fcb17fff640 1 -- 172.21.15.112:0/625686584 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb24001ce0 con 0x7fcb3014f6b0 2024-03-29T11:00:46.784 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.801+0000 7fcb17fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:46.784 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.801+0000 7fcb2d7fa640 4 set_mon_vals no callback set 2024-03-29T11:00:46.784 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.802+0000 7fcb17fff640 1 -- 172.21.15.112:0/625686584 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fcb24003790 con 0x7fcb3014f6b0 2024-03-29T11:00:46.784 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.802+0000 7fcb17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:46.784 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.802+0000 7fcb17fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:46.784 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.802+0000 7fcb17fff640 10 monclient: dump: 2024-03-29T11:00:46.784 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:46.784 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:46.784 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:46.784 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:46.784 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:46.784 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:46.784 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:46.785 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:46.785 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:46.785 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:46.785 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.803+0000 7fcb35944640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:46.785 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.803+0000 7fcb35944640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:46.785 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.803+0000 7fcb35944640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:46.785 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.803+0000 7fcb35944640 1 -- 172.21.15.112:0/625686584 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb10005080 con 0x7fcb3014f6b0 2024-03-29T11:00:46.787 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.804+0000 7fcb17fff640 1 -- 172.21.15.112:0/625686584 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fcb240038f0 con 0x7fcb3014f6b0 2024-03-29T11:00:46.787 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.805+0000 7fcb17fff640 1 -- 172.21.15.112:0/625686584 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7fcb240848d0 con 0x7fcb3014f6b0 2024-03-29T11:00:46.787 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.805+0000 7fcb17fff640 10 client.4613.objecter ms_dispatch 0x7fcb30000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:46.787 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.805+0000 7fcb17fff640 3 client.4613.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:46.787 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.805+0000 7fcb17fff640 3 client.4613.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:46.787 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.805+0000 7fcb17fff640 20 client.4613.objecter dump_active .. 0 homeless 2024-03-29T11:00:46.787 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.806+0000 7fcb2effd640 10 monclient: get_auth_request con 0x7fcb1c06df90 auth_method 0 2024-03-29T11:00:46.787 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.807+0000 7fcb17fff640 10 client.4613.objecter ms_handle_connect 0x7fcb1c06df90 2024-03-29T11:00:46.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.813+0000 7fcb17fff640 1 -- 172.21.15.112:0/625686584 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcb2404ffe0 con 0x7fcb3014f6b0 2024-03-29T11:00:46.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.813+0000 7fcb17fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:46.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:46.813+0000 7fcb17fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:47.134 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:47.154+0000 7fcb35944640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:47.134 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:47.154+0000 7fcb35944640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:47.134 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:47.154+0000 7fcb35944640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:00:47.134 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:47.154+0000 7fcb35944640 1 -- 172.21.15.112:0/625686584 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb10005330 con 0x7fcb3014f6b0 2024-03-29T11:00:47.137 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:47.157+0000 7fcb17fff640 1 -- 172.21.15.112:0/625686584 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7fcb240520e0 con 0x7fcb3014f6b0 2024-03-29T11:00:47.137 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:47.157+0000 7fcb17fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:47.138 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:47.157+0000 7fcb17fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:47.141 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T11:00:47.141 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:47.142 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:47.161+0000 7fcb35944640 10 librados: watch_flush enter 2024-03-29T11:00:47.142 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:47.162+0000 7fcb35944640 10 librados: watch_flush exit 2024-03-29T11:00:47.142 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:47.162+0000 7fcb35944640 20 client.4613.objecter shutdown clearing up homeless session... 2024-03-29T11:00:47.143 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:47.162+0000 7fcb35944640 10 client.4613.objecter successfully canceled tick 2024-03-29T11:00:47.143 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:47.162+0000 7fcb35944640 1 -- 172.21.15.112:0/625686584 >> v1:172.21.15.118:6816/3181073408 conn(0x7fcb1c06df90 legacy=0x7fcb1c070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:47.143 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:47.162+0000 7fcb35944640 10 monclient: shutdown 2024-03-29T11:00:47.143 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:47.162+0000 7fcb35944640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:47.143 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:47.162+0000 7fcb35944640 1 -- 172.21.15.112:0/625686584 >> v1:172.21.15.112:6790/0 conn(0x7fcb3014f6b0 legacy=0x7fcb301e3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:47.143 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:47.163+0000 7fcb35944640 1 -- 172.21.15.112:0/625686584 shutdown_connections 2024-03-29T11:00:47.143 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:47.163+0000 7fcb35944640 1 -- 172.21.15.112:0/625686584 >> 172.21.15.112:0/625686584 conn(0x7fcb300b35e0 msgr2=0x7fcb300ad240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:47.144 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:47.164+0000 7fcb35944640 1 -- 172.21.15.112:0/625686584 shutdown_connections 2024-03-29T11:00:47.144 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:47.164+0000 7fcb35944640 1 -- 172.21.15.112:0/625686584 wait complete. 2024-03-29T11:00:47.144 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:47.164+0000 7fcb35944640 1 librados: shutdown 2024-03-29T11:00:47.165 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:53.971 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:54.117 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.136+0000 7f6c8b160640 10 monclient: get_monmap_and_config 2024-03-29T11:00:54.117 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.136+0000 7f6c8b160640 10 monclient: build_initial_monmap 2024-03-29T11:00:54.117 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.136+0000 7f6c8b160640 10 monclient: monmap: 2024-03-29T11:00:54.117 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:00:54.117 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:54.117 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T11:00:54.137823+0000 2024-03-29T11:00:54.117 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T11:00:54.137823+0000 2024-03-29T11:00:54.117 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:54.117 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:54.117 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:00:54.117 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T11:00:54.117 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T11:00:54.118 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:54.118 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.139+0000 7f6c8b160640 1 Processor -- start 2024-03-29T11:00:54.119 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.139+0000 7f6c8b160640 1 -- start start 2024-03-29T11:00:54.119 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.139+0000 7f6c8b160640 10 monclient: init 2024-03-29T11:00:54.120 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.140+0000 7f6c8b160640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:54.120 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.140+0000 7f6c8b160640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:00:54.120 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.140+0000 7f6c8b160640 10 monclient(hunting): picked mon.noname-a con 0x7f6c840b4920 addr v1:172.21.15.112:6789/0 2024-03-29T11:00:54.120 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.140+0000 7f6c8b160640 10 monclient(hunting): picked mon.noname-c con 0x7f6c840b4200 addr v1:172.21.15.118:6789/0 2024-03-29T11:00:54.120 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.140+0000 7f6c8b160640 10 monclient(hunting): picked mon.noname-b con 0x7f6c8413f9a0 addr v1:172.21.15.112:6790/0 2024-03-29T11:00:54.120 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.140+0000 7f6c8b160640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c840b0930 con 0x7f6c840b4920 2024-03-29T11:00:54.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.140+0000 7f6c8b160640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c840af460 con 0x7f6c840b4200 2024-03-29T11:00:54.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.140+0000 7f6c8b160640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c840aeda0 con 0x7f6c8413f9a0 2024-03-29T11:00:54.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.140+0000 7f6c8b160640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:54.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.140+0000 7f6c8b160640 10 monclient(hunting): authenticate will time out at 1409.444549s 2024-03-29T11:00:54.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.141+0000 7f6c8a95f640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f6c8413f9a0 0x7f6c8414d8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:48744/0 (socket says 172.21.15.112:48744) 2024-03-29T11:00:54.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.141+0000 7f6c8a95f640 1 -- 172.21.15.112:0/2110311665 learned_addr learned my addr 172.21.15.112:0/2110311665 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:00:54.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.144+0000 7f6c8915c640 1 -- 172.21.15.112:0/2110311665 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543558017 0 0) 0x7f6c840b0930 con 0x7f6c840b4920 2024-03-29T11:00:54.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.144+0000 7f6c8915c640 10 monclient(hunting): _negotiate 2024-03-29T11:00:54.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.144+0000 7f6c8915c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:54.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.144+0000 7f6c8915c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:54.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.144+0000 7f6c8915c640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:00:54.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.144+0000 7f6c8915c640 1 -- 172.21.15.112:0/2110311665 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c70003520 con 0x7f6c840b4920 2024-03-29T11:00:54.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.144+0000 7f6c8915c640 1 -- 172.21.15.112:0/2110311665 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 899328877 0 0) 0x7f6c70003520 con 0x7f6c840b4920 2024-03-29T11:00:54.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.144+0000 7f6c8915c640 1 -- 172.21.15.112:0/2110311665 >> v1:172.21.15.112:6790/0 conn(0x7f6c8413f9a0 legacy=0x7f6c8414d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:54.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.144+0000 7f6c8915c640 1 -- 172.21.15.112:0/2110311665 >> v1:172.21.15.118:6789/0 conn(0x7f6c840b4200 legacy=0x7f6c840b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:54.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.144+0000 7f6c8915c640 10 monclient: _finish_hunting 0 2024-03-29T11:00:54.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.144+0000 7f6c8915c640 1 monclient: found mon.noname-a 2024-03-29T11:00:54.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.144+0000 7f6c8915c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:54.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.144+0000 7f6c8915c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T11:00:54.125 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.144+0000 7f6c8915c640 1 -- 172.21.15.112:0/2110311665 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c840af980 con 0x7f6c840b4920 2024-03-29T11:00:54.126 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.144+0000 7f6c8915c640 10 monclient: _finish_auth 0 2024-03-29T11:00:54.126 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.144+0000 7f6c8915c640 10 monclient: _check_auth_tickets 2024-03-29T11:00:54.126 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.144+0000 7f6c8915c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:54.126 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.144+0000 7f6c8915c640 1 -- 172.21.15.112:0/2110311665 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f6c74001690 con 0x7f6c840b4920 2024-03-29T11:00:54.126 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.144+0000 7f6c8b160640 5 monclient: authenticate success, global_id 4554 2024-03-29T11:00:54.126 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.144+0000 7f6c8b160640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:54.126 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.144+0000 7f6c8915c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:54.126 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.144+0000 7f6c8915c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:00:54.126 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.144+0000 7f6c8915c640 10 monclient: dump: 2024-03-29T11:00:54.126 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:54.126 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:54.126 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:54.126 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:54.126 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:54.127 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:54.127 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:54.127 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:54.127 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:54.127 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:54.127 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.144+0000 7f6c8b160640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:54.127 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.145+0000 7f6c8915c640 1 -- 172.21.15.112:0/2110311665 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c740017f0 con 0x7f6c840b4920 2024-03-29T11:00:54.127 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.145+0000 7f6c8915c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:54.127 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.145+0000 7f6c8915c640 1 -- 172.21.15.112:0/2110311665 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f6c74001c40 con 0x7f6c840b4920 2024-03-29T11:00:54.127 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.145+0000 7f6c8915c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:54.127 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.145+0000 7f6c8915c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:54.127 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.145+0000 7f6c8915c640 10 monclient: dump: 2024-03-29T11:00:54.127 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:54.127 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:54.127 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:54.128 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:54.128 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:54.128 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:54.128 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:54.128 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:54.128 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:54.128 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:54.128 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.145+0000 7f6c8b160640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:54.128 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.145+0000 7f6c8b160640 4 set_mon_vals no callback set 2024-03-29T11:00:54.129 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.145+0000 7f6c8b160640 10 monclient: shutdown 2024-03-29T11:00:54.129 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.145+0000 7f6c8b160640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:54.129 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.145+0000 7f6c8b160640 1 -- 172.21.15.112:0/2110311665 >> v1:172.21.15.112:6789/0 conn(0x7f6c840b4920 legacy=0x7f6c8413f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:54.129 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.146+0000 7f6c8b160640 1 -- 172.21.15.112:0/2110311665 shutdown_connections 2024-03-29T11:00:54.129 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.146+0000 7f6c8b160640 1 -- 172.21.15.112:0/2110311665 >> 172.21.15.112:0/2110311665 conn(0x7f6c8406d230 msgr2=0x7f6c84066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:54.129 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.146+0000 7f6c8b160640 1 -- 172.21.15.112:0/2110311665 shutdown_connections 2024-03-29T11:00:54.129 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.147+0000 7f6c8b160640 1 -- 172.21.15.112:0/2110311665 wait complete. 2024-03-29T11:00:54.131 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.148+0000 7f6c8b160640 10 monclient: build_initial_monmap 2024-03-29T11:00:54.131 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.148+0000 7f6c8b160640 10 monclient: monmap: 2024-03-29T11:00:54.131 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:00:54.131 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:54.131 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T11:00:54.131 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T11:00:54.132 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:54.132 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:54.132 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:00:54.132 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T11:00:54.132 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T11:00:54.132 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:54.132 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.149+0000 7f6c8b160640 1 librados: starting msgr at 2024-03-29T11:00:54.132 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.149+0000 7f6c8b160640 1 librados: starting objecter 2024-03-29T11:00:54.132 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.149+0000 7f6c8b160640 1 Processor -- start 2024-03-29T11:00:54.132 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.150+0000 7f6c8b160640 1 -- start start 2024-03-29T11:00:54.132 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.150+0000 7f6c8b160640 1 librados: setting wanted keys 2024-03-29T11:00:54.132 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.150+0000 7f6c8b160640 1 librados: calling monclient init 2024-03-29T11:00:54.132 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.150+0000 7f6c8b160640 10 monclient: init 2024-03-29T11:00:54.132 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.150+0000 7f6c8b160640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:54.132 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.150+0000 7f6c8b160640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:00:54.134 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.153+0000 7f6c8b160640 10 monclient(hunting): picked mon.noname-b con 0x7f6c840b4200 addr v1:172.21.15.118:6789/0 2024-03-29T11:00:54.134 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.153+0000 7f6c8a15e640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f6c840b4200 0x7f6c841e4e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:51475/0 (socket says 172.21.15.112:51475) 2024-03-29T11:00:54.134 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.153+0000 7f6c8a15e640 1 -- 172.21.15.112:0/2641796153 learned_addr learned my addr 172.21.15.112:0/2641796153 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:00:54.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.153+0000 7f6c7affd640 10 client.?.objecter ms_handle_connect 0x7f6c840b4200 2024-03-29T11:00:54.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.153+0000 7f6c7affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:54.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.153+0000 7f6c8b160640 10 monclient(hunting): picked mon.noname-c con 0x7f6c8413f9a0 addr v1:172.21.15.112:6790/0 2024-03-29T11:00:54.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.153+0000 7f6c8b160640 10 monclient(hunting): picked mon.noname-a con 0x7f6c84144b10 addr v1:172.21.15.112:6789/0 2024-03-29T11:00:54.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.153+0000 7f6c8b160640 1 -- 172.21.15.112:0/2641796153 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c8406e1e0 con 0x7f6c84144b10 2024-03-29T11:00:54.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.153+0000 7f6c8b160640 1 -- 172.21.15.112:0/2641796153 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c841e48f0 con 0x7f6c840b4200 2024-03-29T11:00:54.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.153+0000 7f6c8b160640 1 -- 172.21.15.112:0/2641796153 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c841e5520 con 0x7f6c8413f9a0 2024-03-29T11:00:54.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.153+0000 7f6c8b160640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:54.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.153+0000 7f6c8b160640 10 monclient(hunting): authenticate will time out at 1409.458124s 2024-03-29T11:00:54.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.154+0000 7f6c7affd640 1 -- 172.21.15.112:0/2641796153 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1145938172 0 0) 0x7f6c841e48f0 con 0x7f6c840b4200 2024-03-29T11:00:54.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.154+0000 7f6c7affd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:54.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.154+0000 7f6c7affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:54.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.154+0000 7f6c7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:54.135 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.154+0000 7f6c7affd640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:00:54.136 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.154+0000 7f6c7affd640 1 -- 172.21.15.112:0/2641796153 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c68004eb0 con 0x7f6c840b4200 2024-03-29T11:00:54.136 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.154+0000 7f6c7affd640 10 client.?.objecter ms_handle_connect 0x7f6c8413f9a0 2024-03-29T11:00:54.136 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.154+0000 7f6c7affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:54.136 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.154+0000 7f6c7affd640 1 -- 172.21.15.112:0/2641796153 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2790702401 0 0) 0x7f6c68004eb0 con 0x7f6c840b4200 2024-03-29T11:00:54.136 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.155+0000 7f6c7affd640 1 -- 172.21.15.112:0/2641796153 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c841e48f0 con 0x7f6c840b4200 2024-03-29T11:00:54.136 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.155+0000 7f6c7affd640 1 -- 172.21.15.112:0/2641796153 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f6c74001c00 con 0x7f6c840b4200 2024-03-29T11:00:54.136 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.155+0000 7f6c7affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:54.136 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.155+0000 7f6c7affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:54.136 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.155+0000 7f6c7affd640 10 monclient(hunting): dump: 2024-03-29T11:00:54.136 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:54.136 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:54.136 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:54.136 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:54.136 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:54.137 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:54.137 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:54.137 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:54.137 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:54.137 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:54.137 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.155+0000 7f6c7affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:54.138 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.155+0000 7f6c7affd640 1 -- 172.21.15.112:0/2641796153 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 541573486 0 0) 0x7f6c841e48f0 con 0x7f6c840b4200 2024-03-29T11:00:54.138 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.155+0000 7f6c7affd640 1 -- 172.21.15.112:0/2641796153 >> v1:172.21.15.112:6790/0 conn(0x7f6c8413f9a0 legacy=0x7f6c84141ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:54.138 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.155+0000 7f6c7affd640 1 -- 172.21.15.112:0/2641796153 >> v1:172.21.15.112:6789/0 conn(0x7f6c84144b10 legacy=0x7f6c841e3af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:00:54.138 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c7affd640 10 monclient: _finish_hunting 0 2024-03-29T11:00:54.138 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c7affd640 1 monclient: found mon.b 2024-03-29T11:00:54.138 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:54.138 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c7affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:00:54.139 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c7affd640 1 -- 172.21.15.112:0/2641796153 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c841e56b0 con 0x7f6c840b4200 2024-03-29T11:00:54.139 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c7affd640 10 monclient: _finish_auth 0 2024-03-29T11:00:54.139 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c7affd640 10 monclient: _check_auth_tickets 2024-03-29T11:00:54.139 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:54.139 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c7affd640 10 client.?.objecter ms_handle_connect 0x7f6c84144b10 2024-03-29T11:00:54.139 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c7affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:54.139 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c8b160640 5 monclient: authenticate success, global_id 4537 2024-03-29T11:00:54.139 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c8b160640 10 monclient: _renew_subs 2024-03-29T11:00:54.139 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c8b160640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:00:54.139 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c8b160640 1 -- 172.21.15.112:0/2641796153 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c841423b0 con 0x7f6c840b4200 2024-03-29T11:00:54.139 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c7affd640 1 -- 172.21.15.112:0/2641796153 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c74003960 con 0x7f6c840b4200 2024-03-29T11:00:54.139 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c7affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:54.139 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c7affd640 1 -- 172.21.15.112:0/2641796153 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f6c74003db0 con 0x7f6c840b4200 2024-03-29T11:00:54.139 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:54.140 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c7affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:54.140 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c7affd640 10 monclient: dump: 2024-03-29T11:00:54.140 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:00:54.140 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:00:54.140 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:54.140 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:00:54.140 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:54.140 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:00:54.140 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:00:54.140 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:00:54.140 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:00:54.140 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:00:54.140 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c8b160640 10 client.4537.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:54.140 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c8b160640 10 monclient: _renew_subs 2024-03-29T11:00:54.140 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c8b160640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:00:54.141 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c8b160640 1 -- 172.21.15.112:0/2641796153 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c84141070 con 0x7f6c840b4200 2024-03-29T11:00:54.141 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c8b160640 1 librados: init done 2024-03-29T11:00:54.141 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.156+0000 7f6c7bfff640 4 set_mon_vals no callback set 2024-03-29T11:00:54.141 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.158+0000 7f6c8b160640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:54.141 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.158+0000 7f6c8b160640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:54.141 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.158+0000 7f6c8b160640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:00:54.141 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.158+0000 7f6c8b160640 1 -- 172.21.15.112:0/2641796153 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c841e4a80 con 0x7f6c840b4200 2024-03-29T11:00:54.142 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.160+0000 7f6c7affd640 1 -- 172.21.15.112:0/2641796153 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f6c74003f10 con 0x7f6c840b4200 2024-03-29T11:00:54.142 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.161+0000 7f6c7affd640 1 -- 172.21.15.112:0/2641796153 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f6c74084bb0 con 0x7f6c840b4200 2024-03-29T11:00:54.142 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.161+0000 7f6c7affd640 10 client.4537.objecter ms_dispatch 0x7f6c84000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:54.142 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.161+0000 7f6c7affd640 3 client.4537.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:54.142 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.161+0000 7f6c7affd640 3 client.4537.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:54.142 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.161+0000 7f6c7affd640 20 client.4537.objecter dump_active .. 0 homeless 2024-03-29T11:00:54.142 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.161+0000 7f6c8995d640 10 monclient: get_auth_request con 0x7f6c6806dce0 auth_method 0 2024-03-29T11:00:54.142 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.162+0000 7f6c7affd640 10 client.4537.objecter ms_handle_connect 0x7f6c6806dce0 2024-03-29T11:00:54.147 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.167+0000 7f6c7affd640 1 -- 172.21.15.112:0/2641796153 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6c74050490 con 0x7f6c840b4200 2024-03-29T11:00:54.147 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.167+0000 7f6c7affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:54.148 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.167+0000 7f6c7affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:54.486 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.505+0000 7f6c8b160640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:54.486 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.505+0000 7f6c8b160640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:54.486 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.505+0000 7f6c8b160640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:00:54.486 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.505+0000 7f6c8b160640 1 -- 172.21.15.112:0/2641796153 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c840b08c0 con 0x7f6c840b4200 2024-03-29T11:00:54.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.506+0000 7f6c7affd640 1 -- 172.21.15.112:0/2641796153 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f6c740523c0 con 0x7f6c840b4200 2024-03-29T11:00:54.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.506+0000 7f6c7affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:54.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.506+0000 7f6c7affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:54.487 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T11:00:54.487 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:54.490 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.510+0000 7f6c8b160640 10 librados: watch_flush enter 2024-03-29T11:00:54.490 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.510+0000 7f6c8b160640 10 librados: watch_flush exit 2024-03-29T11:00:54.490 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.510+0000 7f6c8b160640 20 client.4537.objecter shutdown clearing up homeless session... 2024-03-29T11:00:54.490 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.510+0000 7f6c8b160640 10 client.4537.objecter successfully canceled tick 2024-03-29T11:00:54.491 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.510+0000 7f6c8b160640 1 -- 172.21.15.112:0/2641796153 >> v1:172.21.15.118:6816/3181073408 conn(0x7f6c6806dce0 legacy=0x7f6c680701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:54.491 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.510+0000 7f6c8b160640 10 monclient: shutdown 2024-03-29T11:00:54.491 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.510+0000 7f6c8b160640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:54.491 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.510+0000 7f6c8b160640 1 -- 172.21.15.112:0/2641796153 >> v1:172.21.15.118:6789/0 conn(0x7f6c840b4200 legacy=0x7f6c841e4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:54.491 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.511+0000 7f6c8b160640 1 -- 172.21.15.112:0/2641796153 shutdown_connections 2024-03-29T11:00:54.491 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.511+0000 7f6c8b160640 1 -- 172.21.15.112:0/2641796153 >> 172.21.15.112:0/2641796153 conn(0x7f6c8406d230 msgr2=0x7f6c84066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:54.492 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.511+0000 7f6c8b160640 1 -- 172.21.15.112:0/2641796153 shutdown_connections 2024-03-29T11:00:54.492 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.511+0000 7f6c8b160640 1 -- 172.21.15.112:0/2641796153 wait complete. 2024-03-29T11:00:54.492 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:00:54.511+0000 7f6c8b160640 1 librados: shutdown 2024-03-29T11:00:54.509 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:01.316 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:01.465 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.485+0000 7f1af4f25640 10 monclient: get_monmap_and_config 2024-03-29T11:01:01.465 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.485+0000 7f1af4f25640 10 monclient: build_initial_monmap 2024-03-29T11:01:01.465 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.485+0000 7f1af4f25640 10 monclient: monmap: 2024-03-29T11:01:01.465 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:01:01.465 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:01.465 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T11:01:01.486187+0000 2024-03-29T11:01:01.465 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T11:01:01.486187+0000 2024-03-29T11:01:01.465 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:01.465 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:01.465 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:01:01.465 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T11:01:01.466 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T11:01:01.466 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:01.466 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.486+0000 7f1af4f25640 1 Processor -- start 2024-03-29T11:01:01.466 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.486+0000 7f1af4f25640 1 -- start start 2024-03-29T11:01:01.466 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.486+0000 7f1af4f25640 10 monclient: init 2024-03-29T11:01:01.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.487+0000 7f1af4f25640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:01.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.487+0000 7f1af4f25640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:01:01.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.487+0000 7f1af4f25640 10 monclient(hunting): picked mon.noname-c con 0x7f1af01422b0 addr v1:172.21.15.118:6789/0 2024-03-29T11:01:01.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.487+0000 7f1af4f25640 10 monclient(hunting): picked mon.noname-b con 0x7f1af0142690 addr v1:172.21.15.112:6790/0 2024-03-29T11:01:01.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.487+0000 7f1af4f25640 10 monclient(hunting): picked mon.noname-a con 0x7f1af00b17d0 addr v1:172.21.15.112:6789/0 2024-03-29T11:01:01.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.487+0000 7f1af4f25640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af0142c80 con 0x7f1af00b17d0 2024-03-29T11:01:01.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.487+0000 7f1af4f25640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af00aae40 con 0x7f1af01422b0 2024-03-29T11:01:01.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.487+0000 7f1af4f25640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af013fbd0 con 0x7f1af0142690 2024-03-29T11:01:01.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.487+0000 7f1af4f25640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:01.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.487+0000 7f1af4f25640 10 monclient(hunting): authenticate will time out at 1416.791605s 2024-03-29T11:01:01.469 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.487+0000 7f1aedd9b640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f1af0142690 0x7f1af00b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:50268/0 (socket says 172.21.15.112:50268) 2024-03-29T11:01:01.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.487+0000 7f1aedd9b640 1 -- 172.21.15.112:0/268346113 learned_addr learned my addr 172.21.15.112:0/268346113 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:01:01.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 1 -- 172.21.15.112:0/268346113 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677499091 0 0) 0x7f1af00aae40 con 0x7f1af01422b0 2024-03-29T11:01:01.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 10 monclient(hunting): _negotiate 2024-03-29T11:01:01.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:01.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:01.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:01:01.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 1 -- 172.21.15.112:0/268346113 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad8003520 con 0x7f1af01422b0 2024-03-29T11:01:01.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 1 -- 172.21.15.112:0/268346113 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653907196 0 0) 0x7f1af013fbd0 con 0x7f1af0142690 2024-03-29T11:01:01.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 10 monclient(hunting): _negotiate 2024-03-29T11:01:01.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:01.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:01.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:01:01.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 1 -- 172.21.15.112:0/268346113 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af00aae40 con 0x7f1af0142690 2024-03-29T11:01:01.470 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 1 -- 172.21.15.112:0/268346113 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 714932518 0 0) 0x7f1ad8003520 con 0x7f1af01422b0 2024-03-29T11:01:01.471 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 1 -- 172.21.15.112:0/268346113 >> v1:172.21.15.112:6790/0 conn(0x7f1af0142690 legacy=0x7f1af00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:01.471 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 1 -- 172.21.15.112:0/268346113 >> v1:172.21.15.112:6789/0 conn(0x7f1af00b17d0 legacy=0x7f1af00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:01.471 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 10 monclient: _finish_hunting 0 2024-03-29T11:01:01.471 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 1 monclient: found mon.noname-c 2024-03-29T11:01:01.471 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:01.471 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T11:01:01.471 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 1 -- 172.21.15.112:0/268346113 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af0151940 con 0x7f1af01422b0 2024-03-29T11:01:01.471 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 10 monclient: _finish_auth 0 2024-03-29T11:01:01.471 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 10 monclient: _check_auth_tickets 2024-03-29T11:01:01.471 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:01.471 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 1 -- 172.21.15.112:0/268346113 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1ae0001690 con 0x7f1af01422b0 2024-03-29T11:01:01.471 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:01.471 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:01.471 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 10 monclient: dump: 2024-03-29T11:01:01.471 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:01.472 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:01.472 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:01.472 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:01.472 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:01.472 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:01.472 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:01.472 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:01.472 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:01.472 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:01.472 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1af4f25640 5 monclient: authenticate success, global_id 4540 2024-03-29T11:01:01.472 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1af4f25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:01.472 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 1 -- 172.21.15.112:0/268346113 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ae00017f0 con 0x7f1af01422b0 2024-03-29T11:01:01.472 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:01.472 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1af4f25640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:01.473 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 1 -- 172.21.15.112:0/268346113 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1ae0001b80 con 0x7f1af01422b0 2024-03-29T11:01:01.473 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:01.473 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1af4f25640 4 set_mon_vals no callback set 2024-03-29T11:01:01.473 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:01.473 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.488+0000 7f1aed59a640 10 monclient: dump: 2024-03-29T11:01:01.473 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:01.473 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:01.473 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:01.473 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:01.473 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:01.473 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:01.473 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:01.473 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:01.473 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:01.473 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:01.479 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.489+0000 7f1af4f25640 10 monclient: shutdown 2024-03-29T11:01:01.479 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.489+0000 7f1af4f25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:01.479 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.489+0000 7f1af4f25640 1 -- 172.21.15.112:0/268346113 >> v1:172.21.15.118:6789/0 conn(0x7f1af01422b0 legacy=0x7f1af013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:01.479 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.489+0000 7f1af4f25640 1 -- 172.21.15.112:0/268346113 shutdown_connections 2024-03-29T11:01:01.479 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.489+0000 7f1af4f25640 1 -- 172.21.15.112:0/268346113 >> 172.21.15.112:0/268346113 conn(0x7f1af0066a00 msgr2=0x7f1af0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:01.479 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.490+0000 7f1af4f25640 1 -- 172.21.15.112:0/268346113 shutdown_connections 2024-03-29T11:01:01.479 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.490+0000 7f1af4f25640 1 -- 172.21.15.112:0/268346113 wait complete. 2024-03-29T11:01:01.479 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.491+0000 7f1af4f25640 10 monclient: build_initial_monmap 2024-03-29T11:01:01.479 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.491+0000 7f1af4f25640 10 monclient: monmap: 2024-03-29T11:01:01.479 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:01:01.479 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:01.480 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T11:01:01.480 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T11:01:01.480 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:01.480 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:01.480 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:01:01.480 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T11:01:01.480 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T11:01:01.480 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:01.480 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.491+0000 7f1af4f25640 1 librados: starting msgr at 2024-03-29T11:01:01.480 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.491+0000 7f1af4f25640 1 librados: starting objecter 2024-03-29T11:01:01.480 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.491+0000 7f1af4f25640 1 Processor -- start 2024-03-29T11:01:01.480 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.491+0000 7f1af4f25640 1 -- start start 2024-03-29T11:01:01.480 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.491+0000 7f1af4f25640 1 librados: setting wanted keys 2024-03-29T11:01:01.480 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.491+0000 7f1af4f25640 1 librados: calling monclient init 2024-03-29T11:01:01.481 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.491+0000 7f1af4f25640 10 monclient: init 2024-03-29T11:01:01.481 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.492+0000 7f1af4f25640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:01.481 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.492+0000 7f1af4f25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:01:01.481 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.492+0000 7f1af4f25640 10 monclient(hunting): picked mon.noname-a con 0x7f1af00b17d0 addr v1:172.21.15.112:6789/0 2024-03-29T11:01:01.481 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.492+0000 7f1af4f25640 10 monclient(hunting): picked mon.noname-b con 0x7f1af01422b0 addr v1:172.21.15.118:6789/0 2024-03-29T11:01:01.481 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.492+0000 7f1af4f25640 10 monclient(hunting): picked mon.noname-c con 0x7f1af0142690 addr v1:172.21.15.112:6790/0 2024-03-29T11:01:01.481 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.492+0000 7f1af4f25640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af00b1010 con 0x7f1af00b17d0 2024-03-29T11:01:01.481 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.492+0000 7f1af4f25640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af00b12f0 con 0x7f1af01422b0 2024-03-29T11:01:01.481 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.492+0000 7f1af4f25640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af00afe90 con 0x7f1af0142690 2024-03-29T11:01:01.481 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.492+0000 7f1af4f25640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:01.481 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.492+0000 7f1af4f25640 10 monclient(hunting): authenticate will time out at 1416.797030s 2024-03-29T11:01:01.481 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1aee59c640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f1af00b17d0 0x7f1af00af170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:52134/0 (socket says 172.21.15.112:52134) 2024-03-29T11:01:01.481 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1aee59c640 1 -- 172.21.15.112:0/1277055457 learned_addr learned my addr 172.21.15.112:0/1277055457 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:01:01.481 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1aeed9d640 1 --1- 172.21.15.112:0/1277055457 >> v1:172.21.15.112:6790/0 conn(0x7f1af0142690 0x7f1af01daa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:51058/0 (socket says 172.21.15.112:51058) 2024-03-29T11:01:01.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1adeffd640 10 client.?.objecter ms_handle_connect 0x7f1af00b17d0 2024-03-29T11:01:01.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1adeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:01.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1adeffd640 10 client.?.objecter ms_handle_connect 0x7f1af0142690 2024-03-29T11:01:01.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1adeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:01.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1adeffd640 10 client.?.objecter ms_handle_connect 0x7f1af01422b0 2024-03-29T11:01:01.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1adeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:01.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981859425 0 0) 0x7f1af00b1010 con 0x7f1af00b17d0 2024-03-29T11:01:01.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1adeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:01.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1adeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:01.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1adeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:01.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1adeffd640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:01:01.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad0004eb0 con 0x7f1af00b17d0 2024-03-29T11:01:01.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763236091 0 0) 0x7f1af00b12f0 con 0x7f1af01422b0 2024-03-29T11:01:01.482 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1adeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:01.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1adeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:01.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1adeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:01.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1adeffd640 10 monclient(hunting): my global_id is 4543 2024-03-29T11:01:01.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af00b1010 con 0x7f1af01422b0 2024-03-29T11:01:01.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 476436347 0 0) 0x7f1af00afe90 con 0x7f1af0142690 2024-03-29T11:01:01.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1adeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:01.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1adeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:01.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1adeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:01.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1adeffd640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:01:01.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.493+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af00b12f0 con 0x7f1af0142690 2024-03-29T11:01:01.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 279888310 0 0) 0x7f1af00b1010 con 0x7f1af01422b0 2024-03-29T11:01:01.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1af00afe90 con 0x7f1af01422b0 2024-03-29T11:01:01.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1ae4001850 con 0x7f1af01422b0 2024-03-29T11:01:01.483 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:01.484 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:01.484 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 10 monclient(hunting): dump: 2024-03-29T11:01:01.484 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:01.484 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:01.484 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:01.484 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:01.484 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:01.484 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:01.484 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:01.484 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:01.484 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:01.484 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:01.484 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:01.484 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 551238055 0 0) 0x7f1af00b12f0 con 0x7f1af0142690 2024-03-29T11:01:01.484 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1af00b1010 con 0x7f1af0142690 2024-03-29T11:01:01.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1078684426 0 0) 0x7f1ad0004eb0 con 0x7f1af00b17d0 2024-03-29T11:01:01.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1af00b12f0 con 0x7f1af00b17d0 2024-03-29T11:01:01.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1ae8001b30 con 0x7f1af0142690 2024-03-29T11:01:01.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:01.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:01.485 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 10 monclient(hunting): dump: 2024-03-29T11:01:01.485 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:01.485 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:01.485 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:01.485 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:01.485 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:01.485 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:01.485 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:01.486 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:01.486 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:01.486 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:01.486 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1ae0001b00 con 0x7f1af00b17d0 2024-03-29T11:01:01.486 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:01.486 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:01.486 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 10 monclient(hunting): dump: 2024-03-29T11:01:01.486 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:01.486 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:01.486 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:01.486 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:01.486 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:01.486 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:01.486 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:01.487 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:01.487 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:01.487 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:01.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2199432237 0 0) 0x7f1af00afe90 con 0x7f1af01422b0 2024-03-29T11:01:01.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 >> v1:172.21.15.112:6790/0 conn(0x7f1af0142690 legacy=0x7f1af01daa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:01.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 >> v1:172.21.15.112:6789/0 conn(0x7f1af00b17d0 legacy=0x7f1af00af170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:01.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:01:01.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 1 monclient: found mon.b 2024-03-29T11:01:01.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:01.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:01:01.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af00b0020 con 0x7f1af01422b0 2024-03-29T11:01:01.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 10 monclient: _finish_auth 0 2024-03-29T11:01:01.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:01:01.487 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1adeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:01.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1af4f25640 5 monclient: authenticate success, global_id 4543 2024-03-29T11:01:01.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1af4f25640 10 monclient: _renew_subs 2024-03-29T11:01:01.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1af4f25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:01:01.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.494+0000 7f1af4f25640 1 -- 172.21.15.112:0/1277055457 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af0137770 con 0x7f1af01422b0 2024-03-29T11:01:01.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.495+0000 7f1af4f25640 10 client.4543.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:01.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.495+0000 7f1af4f25640 10 monclient: _renew_subs 2024-03-29T11:01:01.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.495+0000 7f1af4f25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:01:01.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.495+0000 7f1af4f25640 1 -- 172.21.15.112:0/1277055457 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af01db150 con 0x7f1af01422b0 2024-03-29T11:01:01.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.495+0000 7f1af4f25640 1 librados: init done 2024-03-29T11:01:01.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.496+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ae40047e0 con 0x7f1af01422b0 2024-03-29T11:01:01.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.496+0000 7f1adeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:01.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.496+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1ae4004af0 con 0x7f1af01422b0 2024-03-29T11:01:01.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.496+0000 7f1adeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:01.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.496+0000 7f1aecd99640 4 set_mon_vals no callback set 2024-03-29T11:01:01.488 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.496+0000 7f1af4f25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:01.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.496+0000 7f1adeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:01.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.496+0000 7f1adeffd640 10 monclient: dump: 2024-03-29T11:01:01.489 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:01.489 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:01.489 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:01.489 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:01.489 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:01.489 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:01.489 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:01.489 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:01.489 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:01.489 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:01.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.497+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f1ae401aac0 con 0x7f1af01422b0 2024-03-29T11:01:01.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.497+0000 7f1af4f25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:01.489 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.497+0000 7f1af4f25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:01:01.490 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.497+0000 7f1af4f25640 1 -- 172.21.15.112:0/1277055457 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af00b05b0 con 0x7f1af01422b0 2024-03-29T11:01:01.490 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.498+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f1ae4084910 con 0x7f1af01422b0 2024-03-29T11:01:01.490 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.498+0000 7f1adeffd640 10 client.4543.objecter ms_dispatch 0x7f1af0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:01.490 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.498+0000 7f1adeffd640 3 client.4543.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:01.490 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.498+0000 7f1adeffd640 3 client.4543.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:01.490 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.498+0000 7f1adeffd640 20 client.4543.objecter dump_active .. 0 homeless 2024-03-29T11:01:01.490 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.498+0000 7f1aee59c640 10 monclient: get_auth_request con 0x7f1ad006e1d0 auth_method 0 2024-03-29T11:01:01.490 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.498+0000 7f1adeffd640 10 client.4543.objecter ms_handle_connect 0x7f1ad006e1d0 2024-03-29T11:01:01.491 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.506+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1ae40501f0 con 0x7f1af01422b0 2024-03-29T11:01:01.491 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.506+0000 7f1adeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:01.491 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.506+0000 7f1adeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:01.834 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.854+0000 7f1af4f25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:01.834 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.854+0000 7f1af4f25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:01.834 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.854+0000 7f1af4f25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:01:01.834 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.854+0000 7f1af4f25640 1 -- 172.21.15.112:0/1277055457 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1af0075f10 con 0x7f1af01422b0 2024-03-29T11:01:01.835 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.855+0000 7f1adeffd640 1 -- 172.21.15.112:0/1277055457 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f1ae4052120 con 0x7f1af01422b0 2024-03-29T11:01:01.835 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.856+0000 7f1adeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:01.835 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.856+0000 7f1adeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:01.836 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T11:01:01.836 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:01.839 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.858+0000 7f1af4f25640 10 librados: watch_flush enter 2024-03-29T11:01:01.839 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.858+0000 7f1af4f25640 10 librados: watch_flush exit 2024-03-29T11:01:01.839 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.858+0000 7f1af4f25640 20 client.4543.objecter shutdown clearing up homeless session... 2024-03-29T11:01:01.839 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.858+0000 7f1af4f25640 10 client.4543.objecter successfully canceled tick 2024-03-29T11:01:01.839 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.858+0000 7f1af4f25640 1 -- 172.21.15.112:0/1277055457 >> v1:172.21.15.118:6816/3181073408 conn(0x7f1ad006e1d0 legacy=0x7f1ad0070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:01.839 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.858+0000 7f1af4f25640 10 monclient: shutdown 2024-03-29T11:01:01.839 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.858+0000 7f1af4f25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:01.839 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.858+0000 7f1af4f25640 1 -- 172.21.15.112:0/1277055457 >> v1:172.21.15.118:6789/0 conn(0x7f1af01422b0 legacy=0x7f1af0137060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:01.839 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.858+0000 7f1af4f25640 1 -- 172.21.15.112:0/1277055457 shutdown_connections 2024-03-29T11:01:01.839 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.858+0000 7f1af4f25640 1 -- 172.21.15.112:0/1277055457 >> 172.21.15.112:0/1277055457 conn(0x7f1af0066a00 msgr2=0x7f1af013d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:01.839 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.858+0000 7f1af4f25640 1 -- 172.21.15.112:0/1277055457 shutdown_connections 2024-03-29T11:01:01.839 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.858+0000 7f1af4f25640 1 -- 172.21.15.112:0/1277055457 wait complete. 2024-03-29T11:01:01.840 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:01.859+0000 7f1af4f25640 1 librados: shutdown 2024-03-29T11:01:01.852 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:08.659 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:08.788 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.808+0000 7f5843788640 10 monclient: get_monmap_and_config 2024-03-29T11:01:08.789 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.808+0000 7f5843788640 10 monclient: build_initial_monmap 2024-03-29T11:01:08.789 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.808+0000 7f5843788640 10 monclient: monmap: 2024-03-29T11:01:08.789 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:01:08.789 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:08.789 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T11:01:08.809773+0000 2024-03-29T11:01:08.789 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T11:01:08.809773+0000 2024-03-29T11:01:08.789 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:08.789 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:08.789 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:01:08.789 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T11:01:08.789 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T11:01:08.790 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:08.792 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.812+0000 7f5843788640 1 Processor -- start 2024-03-29T11:01:08.793 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.813+0000 7f5843788640 1 -- start start 2024-03-29T11:01:08.793 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.813+0000 7f5843788640 10 monclient: init 2024-03-29T11:01:08.796 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.814+0000 7f5843788640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:08.796 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.814+0000 7f5843788640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:01:08.796 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.814+0000 7f5843788640 10 monclient(hunting): picked mon.noname-b con 0x7f583c1457f0 addr v1:172.21.15.112:6790/0 2024-03-29T11:01:08.796 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.814+0000 7f5843788640 10 monclient(hunting): picked mon.noname-c con 0x7f583c13f650 addr v1:172.21.15.118:6789/0 2024-03-29T11:01:08.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.814+0000 7f5843788640 10 monclient(hunting): picked mon.noname-a con 0x7f583c0a9d70 addr v1:172.21.15.112:6789/0 2024-03-29T11:01:08.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.814+0000 7f5843788640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f583c0aaef0 con 0x7f583c0a9d70 2024-03-29T11:01:08.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.814+0000 7f5843788640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f583c145bd0 con 0x7f583c13f650 2024-03-29T11:01:08.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.814+0000 7f5843788640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f583c145eb0 con 0x7f583c1457f0 2024-03-29T11:01:08.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.814+0000 7f5843788640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:08.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.814+0000 7f5843788640 10 monclient(hunting): authenticate will time out at 1424.118292s 2024-03-29T11:01:08.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.814+0000 7f5841524640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f583c1457f0 0x7f583c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:45234/0 (socket says 172.21.15.112:45234) 2024-03-29T11:01:08.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.814+0000 7f5841524640 1 -- 172.21.15.112:0/2232452928 learned_addr learned my addr 172.21.15.112:0/2232452928 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:01:08.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.815+0000 7f5833fff640 1 -- 172.21.15.112:0/2232452928 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3978354459 0 0) 0x7f583c145eb0 con 0x7f583c1457f0 2024-03-29T11:01:08.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.815+0000 7f5833fff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:08.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.815+0000 7f5833fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:08.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.815+0000 7f5833fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:08.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.815+0000 7f5833fff640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:01:08.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.815+0000 7f5833fff640 1 -- 172.21.15.112:0/2232452928 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5828003520 con 0x7f583c1457f0 2024-03-29T11:01:08.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.815+0000 7f5833fff640 1 -- 172.21.15.112:0/2232452928 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135776139 0 0) 0x7f583c145bd0 con 0x7f583c13f650 2024-03-29T11:01:08.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.815+0000 7f5833fff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:08.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.815+0000 7f5833fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:08.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.815+0000 7f5833fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:08.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.815+0000 7f5833fff640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:01:08.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.815+0000 7f5833fff640 1 -- 172.21.15.112:0/2232452928 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f583c145eb0 con 0x7f583c13f650 2024-03-29T11:01:08.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.815+0000 7f5833fff640 1 -- 172.21.15.112:0/2232452928 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2365306760 0 0) 0x7f583c145eb0 con 0x7f583c13f650 2024-03-29T11:01:08.799 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.815+0000 7f5833fff640 1 -- 172.21.15.112:0/2232452928 >> v1:172.21.15.112:6790/0 conn(0x7f583c1457f0 legacy=0x7f583c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:08.799 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.815+0000 7f5833fff640 1 -- 172.21.15.112:0/2232452928 >> v1:172.21.15.112:6789/0 conn(0x7f583c0a9d70 legacy=0x7f583c14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:08.807 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.818+0000 7f5833fff640 10 monclient: _finish_hunting 0 2024-03-29T11:01:08.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.818+0000 7f5833fff640 1 monclient: found mon.noname-c 2024-03-29T11:01:08.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.818+0000 7f5833fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:08.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.818+0000 7f5833fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T11:01:08.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.818+0000 7f5833fff640 1 -- 172.21.15.112:0/2232452928 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583c146340 con 0x7f583c13f650 2024-03-29T11:01:08.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.818+0000 7f5833fff640 10 monclient: _finish_auth 0 2024-03-29T11:01:08.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.818+0000 7f5833fff640 10 monclient: _check_auth_tickets 2024-03-29T11:01:08.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.818+0000 7f5833fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:08.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.818+0000 7f5843788640 5 monclient: authenticate success, global_id 4546 2024-03-29T11:01:08.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.818+0000 7f5843788640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:08.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.818+0000 7f5833fff640 1 -- 172.21.15.112:0/2232452928 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f5834001670 con 0x7f583c13f650 2024-03-29T11:01:08.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.818+0000 7f5833fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:08.809 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.818+0000 7f5833fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:08.809 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.818+0000 7f5833fff640 10 monclient: dump: 2024-03-29T11:01:08.809 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:08.809 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:08.809 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:08.809 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:08.809 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:08.809 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:08.809 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:08.809 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:08.809 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:08.810 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:08.810 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.818+0000 7f5843788640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:08.810 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.818+0000 7f5833fff640 1 -- 172.21.15.112:0/2232452928 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58340018c0 con 0x7f583c13f650 2024-03-29T11:01:08.810 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.818+0000 7f5833fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:08.810 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.819+0000 7f5843788640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:08.810 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.819+0000 7f5843788640 4 set_mon_vals no callback set 2024-03-29T11:01:08.810 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.819+0000 7f5843788640 10 monclient: shutdown 2024-03-29T11:01:08.810 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.819+0000 7f5843788640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:08.810 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.819+0000 7f5843788640 1 -- 172.21.15.112:0/2232452928 >> v1:172.21.15.118:6789/0 conn(0x7f583c13f650 legacy=0x7f583c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:08.810 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.820+0000 7f5843788640 1 -- 172.21.15.112:0/2232452928 shutdown_connections 2024-03-29T11:01:08.810 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.820+0000 7f5843788640 1 -- 172.21.15.112:0/2232452928 >> 172.21.15.112:0/2232452928 conn(0x7f583c066a00 msgr2=0x7f583c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:08.810 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.820+0000 7f5843788640 1 -- 172.21.15.112:0/2232452928 shutdown_connections 2024-03-29T11:01:08.811 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.820+0000 7f5843788640 1 -- 172.21.15.112:0/2232452928 wait complete. 2024-03-29T11:01:08.811 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.822+0000 7f5843788640 10 monclient: build_initial_monmap 2024-03-29T11:01:08.811 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.822+0000 7f5843788640 10 monclient: monmap: 2024-03-29T11:01:08.811 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:01:08.811 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:08.811 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T11:01:08.811 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T11:01:08.811 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:08.811 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:08.812 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:01:08.812 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T11:01:08.812 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T11:01:08.812 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:08.812 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.822+0000 7f5843788640 1 librados: starting msgr at 2024-03-29T11:01:08.812 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.822+0000 7f5843788640 1 librados: starting objecter 2024-03-29T11:01:08.812 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.822+0000 7f5843788640 1 Processor -- start 2024-03-29T11:01:08.812 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.822+0000 7f5843788640 1 -- start start 2024-03-29T11:01:08.812 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.822+0000 7f5843788640 1 librados: setting wanted keys 2024-03-29T11:01:08.812 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.822+0000 7f5843788640 1 librados: calling monclient init 2024-03-29T11:01:08.812 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.822+0000 7f5843788640 10 monclient: init 2024-03-29T11:01:08.812 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.823+0000 7f5843788640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:08.813 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.823+0000 7f5843788640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:01:08.813 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.823+0000 7f5843788640 10 monclient(hunting): picked mon.noname-b con 0x7f583c0a9d70 addr v1:172.21.15.118:6789/0 2024-03-29T11:01:08.813 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.823+0000 7f5843788640 10 monclient(hunting): picked mon.noname-c con 0x7f583c13f650 addr v1:172.21.15.112:6790/0 2024-03-29T11:01:08.813 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.823+0000 7f5843788640 10 monclient(hunting): picked mon.noname-a con 0x7f583c1457f0 addr v1:172.21.15.112:6789/0 2024-03-29T11:01:08.813 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.823+0000 7f5843788640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f583c1d6dd0 con 0x7f583c1457f0 2024-03-29T11:01:08.813 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.823+0000 7f5843788640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f583c1387a0 con 0x7f583c0a9d70 2024-03-29T11:01:08.813 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.823+0000 7f5843788640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f583c13fda0 con 0x7f583c13f650 2024-03-29T11:01:08.813 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.823+0000 7f5843788640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:08.813 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.823+0000 7f5843788640 10 monclient(hunting): authenticate will time out at 1424.127526s 2024-03-29T11:01:08.813 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.823+0000 7f5840d23640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f583c13f650 0x7f583c137980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:46748/0 (socket says 172.21.15.112:46748) 2024-03-29T11:01:08.813 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.823+0000 7f5840d23640 1 -- 172.21.15.112:0/3231891182 learned_addr learned my addr 172.21.15.112:0/3231891182 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:01:08.813 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.823+0000 7f5831ffb640 10 client.?.objecter ms_handle_connect 0x7f583c13f650 2024-03-29T11:01:08.813 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.823+0000 7f5831ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:08.814 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.823+0000 7f5831ffb640 1 -- 172.21.15.112:0/3231891182 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455798177 0 0) 0x7f583c13fda0 con 0x7f583c13f650 2024-03-29T11:01:08.814 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.823+0000 7f5831ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:08.814 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.823+0000 7f5831ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:08.814 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.823+0000 7f5831ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:08.814 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.823+0000 7f5831ffb640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:01:08.814 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.823+0000 7f5831ffb640 1 -- 172.21.15.112:0/3231891182 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5820004eb0 con 0x7f583c13f650 2024-03-29T11:01:08.814 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.823+0000 7f5831ffb640 1 -- 172.21.15.112:0/3231891182 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1117610277 0 0) 0x7f5820004eb0 con 0x7f583c13f650 2024-03-29T11:01:08.814 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.823+0000 7f5831ffb640 1 -- 172.21.15.112:0/3231891182 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f583c13fda0 con 0x7f583c13f650 2024-03-29T11:01:08.814 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.824+0000 7f5831ffb640 1 -- 172.21.15.112:0/3231891182 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f5834003cf0 con 0x7f583c13f650 2024-03-29T11:01:08.814 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.824+0000 7f5831ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:08.814 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.824+0000 7f5831ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:08.814 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.824+0000 7f5831ffb640 10 monclient(hunting): dump: 2024-03-29T11:01:08.814 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:08.814 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:08.815 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:08.815 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:08.815 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:08.815 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:08.815 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:08.815 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:08.815 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:08.815 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:08.815 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.824+0000 7f5831ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:08.815 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.824+0000 7f5831ffb640 1 -- 172.21.15.112:0/3231891182 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 275379799 0 0) 0x7f583c13fda0 con 0x7f583c13f650 2024-03-29T11:01:08.815 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.824+0000 7f5831ffb640 1 -- 172.21.15.112:0/3231891182 >> v1:172.21.15.118:6789/0 conn(0x7f583c0a9d70 legacy=0x7f583c1d7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:08.815 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.824+0000 7f5831ffb640 1 -- 172.21.15.112:0/3231891182 >> v1:172.21.15.112:6789/0 conn(0x7f583c1457f0 legacy=0x7f583c138090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:01:08.815 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.825+0000 7f5831ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:01:08.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.825+0000 7f5831ffb640 1 monclient: found mon.c 2024-03-29T11:01:08.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.825+0000 7f5831ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:08.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.825+0000 7f5831ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:01:08.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.825+0000 7f5831ffb640 1 -- 172.21.15.112:0/3231891182 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583c1419d0 con 0x7f583c13f650 2024-03-29T11:01:08.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.825+0000 7f5831ffb640 10 monclient: _finish_auth 0 2024-03-29T11:01:08.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.825+0000 7f5831ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:01:08.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.825+0000 7f5831ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:08.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.825+0000 7f5843788640 5 monclient: authenticate success, global_id 4628 2024-03-29T11:01:08.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.825+0000 7f5843788640 10 monclient: _renew_subs 2024-03-29T11:01:08.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.825+0000 7f5843788640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:01:08.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.825+0000 7f5843788640 1 -- 172.21.15.112:0/3231891182 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f583c141c20 con 0x7f583c13f650 2024-03-29T11:01:08.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.825+0000 7f5843788640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:08.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.825+0000 7f5843788640 10 monclient: _renew_subs 2024-03-29T11:01:08.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.825+0000 7f5843788640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:01:08.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.825+0000 7f5843788640 1 -- 172.21.15.112:0/3231891182 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f583c1d7b50 con 0x7f583c13f650 2024-03-29T11:01:08.817 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.825+0000 7f5843788640 1 librados: init done 2024-03-29T11:01:08.817 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.827+0000 7f5831ffb640 1 -- 172.21.15.112:0/3231891182 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5834001570 con 0x7f583c13f650 2024-03-29T11:01:08.817 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.827+0000 7f5831ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:08.817 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.827+0000 7f5831ffb640 1 -- 172.21.15.112:0/3231891182 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f5834001ab0 con 0x7f583c13f650 2024-03-29T11:01:08.817 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.827+0000 7f5831ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:08.817 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.827+0000 7f5831ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:08.817 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.827+0000 7f58337fe640 4 set_mon_vals no callback set 2024-03-29T11:01:08.817 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.827+0000 7f5831ffb640 10 monclient: dump: 2024-03-29T11:01:08.817 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:08.817 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:08.817 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:08.817 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:08.817 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:08.817 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:08.818 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:08.818 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:08.818 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:08.818 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:08.819 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.828+0000 7f5831ffb640 1 -- 172.21.15.112:0/3231891182 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f5834004100 con 0x7f583c13f650 2024-03-29T11:01:08.819 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.828+0000 7f5843788640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:08.819 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.828+0000 7f5843788640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:08.819 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.828+0000 7f5843788640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:01:08.819 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.828+0000 7f5843788640 1 -- 172.21.15.112:0/3231891182 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f581c005080 con 0x7f583c13f650 2024-03-29T11:01:08.819 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.828+0000 7f5831ffb640 1 -- 172.21.15.112:0/3231891182 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1318866204 0 0) 0x7f5834084d20 con 0x7f583c13f650 2024-03-29T11:01:08.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.828+0000 7f5831ffb640 10 client.4628.objecter ms_dispatch 0x7f583c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:08.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.828+0000 7f5831ffb640 3 client.4628.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:08.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.828+0000 7f5831ffb640 3 client.4628.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:08.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.828+0000 7f5831ffb640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-29T11:01:08.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.829+0000 7f5841524640 10 monclient: get_auth_request con 0x7f582006dc70 auth_method 0 2024-03-29T11:01:08.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.831+0000 7f5831ffb640 10 client.4628.objecter ms_handle_connect 0x7f582006dc70 2024-03-29T11:01:08.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.835+0000 7f5831ffb640 1 -- 172.21.15.112:0/3231891182 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5834050600 con 0x7f583c13f650 2024-03-29T11:01:08.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.835+0000 7f5831ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:08.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:08.835+0000 7f5831ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:09.111 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:09.130+0000 7f5843788640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:09.111 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:09.130+0000 7f5843788640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:09.111 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:09.130+0000 7f5843788640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:01:09.111 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:09.130+0000 7f5843788640 1 -- 172.21.15.112:0/3231891182 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f581c005330 con 0x7f583c13f650 2024-03-29T11:01:09.111 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:09.130+0000 7f5831ffb640 1 -- 172.21.15.112:0/3231891182 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 537467202) 0x7f5834001c10 con 0x7f583c13f650 2024-03-29T11:01:09.111 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:09.131+0000 7f5831ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:09.111 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:09.131+0000 7f5831ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:09.112 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T11:01:09.112 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T10:51:12.183716+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:12.183716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:09.114 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:09.134+0000 7f5843788640 10 librados: watch_flush enter 2024-03-29T11:01:09.114 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:09.134+0000 7f5843788640 10 librados: watch_flush exit 2024-03-29T11:01:09.114 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:09.134+0000 7f5843788640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-29T11:01:09.114 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:09.134+0000 7f5843788640 10 client.4628.objecter successfully canceled tick 2024-03-29T11:01:09.115 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:09.134+0000 7f5843788640 1 -- 172.21.15.112:0/3231891182 >> v1:172.21.15.118:6816/3181073408 conn(0x7f582006dc70 legacy=0x7f5820070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:09.115 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:09.134+0000 7f5843788640 10 monclient: shutdown 2024-03-29T11:01:09.115 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:09.134+0000 7f5843788640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:09.115 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:09.134+0000 7f5843788640 1 -- 172.21.15.112:0/3231891182 >> v1:172.21.15.112:6790/0 conn(0x7f583c13f650 legacy=0x7f583c137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:09.115 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:09.134+0000 7f5843788640 1 -- 172.21.15.112:0/3231891182 shutdown_connections 2024-03-29T11:01:09.115 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:09.134+0000 7f5843788640 1 -- 172.21.15.112:0/3231891182 >> 172.21.15.112:0/3231891182 conn(0x7f583c066a00 msgr2=0x7f583c0b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:09.115 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:09.134+0000 7f5843788640 1 -- 172.21.15.112:0/3231891182 shutdown_connections 2024-03-29T11:01:09.115 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:09.134+0000 7f5843788640 1 -- 172.21.15.112:0/3231891182 wait complete. 2024-03-29T11:01:09.115 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:09.134+0000 7f5843788640 1 librados: shutdown 2024-03-29T11:01:09.130 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:15.937 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:16.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.098+0000 7f92875f4640 10 monclient: get_monmap_and_config 2024-03-29T11:01:16.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.098+0000 7f92875f4640 10 monclient: build_initial_monmap 2024-03-29T11:01:16.078 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.098+0000 7f92875f4640 10 monclient: monmap: 2024-03-29T11:01:16.078 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:01:16.078 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:16.078 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T11:01:16.099177+0000 2024-03-29T11:01:16.079 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T11:01:16.099177+0000 2024-03-29T11:01:16.079 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:16.079 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:16.079 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:01:16.079 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T11:01:16.079 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T11:01:16.079 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:16.079 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.099+0000 7f92875f4640 1 Processor -- start 2024-03-29T11:01:16.084 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.100+0000 7f92875f4640 1 -- start start 2024-03-29T11:01:16.084 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.100+0000 7f92875f4640 10 monclient: init 2024-03-29T11:01:16.084 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.101+0000 7f92875f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:16.084 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.101+0000 7f92875f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:16.084 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.101+0000 7f92875f4640 10 monclient(hunting): picked mon.noname-c con 0x7f92801457f0 addr v1:172.21.15.118:6789/0 2024-03-29T11:01:16.084 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.101+0000 7f92875f4640 10 monclient(hunting): picked mon.noname-a con 0x7f928013f650 addr v1:172.21.15.112:6789/0 2024-03-29T11:01:16.084 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.101+0000 7f92875f4640 10 monclient(hunting): picked mon.noname-b con 0x7f92800a9d70 addr v1:172.21.15.112:6790/0 2024-03-29T11:01:16.085 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.101+0000 7f92875f4640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92800aaef0 con 0x7f928013f650 2024-03-29T11:01:16.085 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.101+0000 7f92875f4640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9280145bd0 con 0x7f92801457f0 2024-03-29T11:01:16.085 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.101+0000 7f92875f4640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9280145eb0 con 0x7f92800a9d70 2024-03-29T11:01:16.085 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.101+0000 7f92875f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:16.085 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.101+0000 7f92875f4640 10 monclient(hunting): authenticate will time out at 1431.405510s 2024-03-29T11:01:16.085 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.103+0000 7f9285390640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f92801457f0 0x7f928013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:45174/0 (socket says 172.21.15.112:45174) 2024-03-29T11:01:16.085 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.103+0000 7f9285390640 1 -- 172.21.15.112:0/2644228016 learned_addr learned my addr 172.21.15.112:0/2644228016 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:01:16.085 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.103+0000 7f9277fff640 1 -- 172.21.15.112:0/2644228016 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2607430269 0 0) 0x7f9280145bd0 con 0x7f92801457f0 2024-03-29T11:01:16.085 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.103+0000 7f9277fff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:16.085 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.103+0000 7f9277fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:16.085 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.103+0000 7f9277fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:16.085 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.103+0000 7f9277fff640 10 monclient(hunting): my global_id is 4549 2024-03-29T11:01:16.085 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.103+0000 7f9277fff640 1 -- 172.21.15.112:0/2644228016 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f926c003520 con 0x7f92801457f0 2024-03-29T11:01:16.086 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.104+0000 7f9277fff640 1 -- 172.21.15.112:0/2644228016 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245572819 0 0) 0x7f9280145eb0 con 0x7f92800a9d70 2024-03-29T11:01:16.086 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.104+0000 7f9277fff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:16.086 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.104+0000 7f9277fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:16.086 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.104+0000 7f9277fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:16.086 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.104+0000 7f9277fff640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:01:16.086 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.104+0000 7f9277fff640 1 -- 172.21.15.112:0/2644228016 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9280145bd0 con 0x7f92800a9d70 2024-03-29T11:01:16.089 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.110+0000 7f9277fff640 1 -- 172.21.15.112:0/2644228016 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2483214375 0 0) 0x7f926c003520 con 0x7f92801457f0 2024-03-29T11:01:16.089 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.110+0000 7f9277fff640 1 -- 172.21.15.112:0/2644228016 >> v1:172.21.15.112:6790/0 conn(0x7f92800a9d70 legacy=0x7f928014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:16.090 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.110+0000 7f9277fff640 1 -- 172.21.15.112:0/2644228016 >> v1:172.21.15.112:6789/0 conn(0x7f928013f650 legacy=0x7f92800b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:16.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.111+0000 7f9277fff640 10 monclient: _finish_hunting 0 2024-03-29T11:01:16.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.111+0000 7f9277fff640 1 monclient: found mon.noname-c 2024-03-29T11:01:16.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.111+0000 7f9277fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:16.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.111+0000 7f9277fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T11:01:16.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.111+0000 7f9277fff640 1 -- 172.21.15.112:0/2644228016 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9280146340 con 0x7f92801457f0 2024-03-29T11:01:16.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.111+0000 7f9277fff640 10 monclient: _finish_auth 0 2024-03-29T11:01:16.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.111+0000 7f9277fff640 10 monclient: _check_auth_tickets 2024-03-29T11:01:16.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.111+0000 7f9277fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:16.091 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.111+0000 7f92875f4640 5 monclient: authenticate success, global_id 4549 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.111+0000 7f9277fff640 1 -- 172.21.15.112:0/2644228016 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9270001690 con 0x7f92801457f0 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.111+0000 7f92875f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.111+0000 7f9277fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.111+0000 7f9277fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.111+0000 7f9277fff640 10 monclient: dump: 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:16.093 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:16.093 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.111+0000 7f92875f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:16.093 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.111+0000 7f9277fff640 1 -- 172.21.15.112:0/2644228016 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9270001870 con 0x7f92801457f0 2024-03-29T11:01:16.093 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.111+0000 7f9277fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:16.093 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.111+0000 7f92875f4640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:16.093 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.111+0000 7f92875f4640 4 set_mon_vals no callback set 2024-03-29T11:01:16.094 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.112+0000 7f9277fff640 1 -- 172.21.15.112:0/2644228016 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f9270001c80 con 0x7f92801457f0 2024-03-29T11:01:16.094 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.112+0000 7f9277fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:16.094 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.112+0000 7f9277fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:16.094 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.112+0000 7f9277fff640 10 monclient: dump: 2024-03-29T11:01:16.094 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.112+0000 7f92875f4640 10 monclient: shutdown 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.112+0000 7f92875f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.112+0000 7f92875f4640 1 -- 172.21.15.112:0/2644228016 >> v1:172.21.15.118:6789/0 conn(0x7f92801457f0 legacy=0x7f928013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.112+0000 7f92875f4640 1 -- 172.21.15.112:0/2644228016 shutdown_connections 2024-03-29T11:01:16.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.112+0000 7f92875f4640 1 -- 172.21.15.112:0/2644228016 >> 172.21.15.112:0/2644228016 conn(0x7f9280066a00 msgr2=0x7f9280066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:16.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.112+0000 7f92875f4640 1 -- 172.21.15.112:0/2644228016 shutdown_connections 2024-03-29T11:01:16.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.112+0000 7f92875f4640 1 -- 172.21.15.112:0/2644228016 wait complete. 2024-03-29T11:01:16.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.113+0000 7f92875f4640 10 monclient: build_initial_monmap 2024-03-29T11:01:16.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.113+0000 7f92875f4640 10 monclient: monmap: 2024-03-29T11:01:16.096 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:01:16.096 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:16.096 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T11:01:16.096 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T11:01:16.096 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:16.096 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:16.096 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:01:16.096 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T11:01:16.096 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T11:01:16.096 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:16.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.114+0000 7f92875f4640 1 librados: starting msgr at 2024-03-29T11:01:16.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.114+0000 7f92875f4640 1 librados: starting objecter 2024-03-29T11:01:16.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.114+0000 7f92875f4640 1 Processor -- start 2024-03-29T11:01:16.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.115+0000 7f92875f4640 1 -- start start 2024-03-29T11:01:16.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.115+0000 7f92875f4640 1 librados: setting wanted keys 2024-03-29T11:01:16.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.115+0000 7f92875f4640 1 librados: calling monclient init 2024-03-29T11:01:16.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.115+0000 7f92875f4640 10 monclient: init 2024-03-29T11:01:16.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.116+0000 7f92875f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:16.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.116+0000 7f92875f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:01:16.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.116+0000 7f92875f4640 10 monclient(hunting): picked mon.noname-c con 0x7f92800a9d70 addr v1:172.21.15.112:6790/0 2024-03-29T11:01:16.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.116+0000 7f92875f4640 10 monclient(hunting): picked mon.noname-b con 0x7f928013f650 addr v1:172.21.15.118:6789/0 2024-03-29T11:01:16.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.116+0000 7f92875f4640 10 monclient(hunting): picked mon.noname-a con 0x7f92801457f0 addr v1:172.21.15.112:6789/0 2024-03-29T11:01:16.098 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.116+0000 7f92875f4640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92801446d0 con 0x7f92801457f0 2024-03-29T11:01:16.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.116+0000 7f92875f4640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9280141300 con 0x7f928013f650 2024-03-29T11:01:16.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.116+0000 7f92875f4640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92801450d0 con 0x7f92800a9d70 2024-03-29T11:01:16.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.116+0000 7f92875f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:16.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.116+0000 7f92875f4640 10 monclient(hunting): authenticate will time out at 1431.420554s 2024-03-29T11:01:16.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.117+0000 7f9284b8f640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f928013f650 0x7f92801d7850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:45260/0 (socket says 172.21.15.112:45260) 2024-03-29T11:01:16.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.117+0000 7f9284b8f640 1 -- 172.21.15.112:0/2816943610 learned_addr learned my addr 172.21.15.112:0/2816943610 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:01:16.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.117+0000 7f9275ffb640 10 client.?.objecter ms_handle_connect 0x7f928013f650 2024-03-29T11:01:16.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.117+0000 7f9275ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:16.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.118+0000 7f9275ffb640 1 -- 172.21.15.112:0/2816943610 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3431961472 0 0) 0x7f9280141300 con 0x7f928013f650 2024-03-29T11:01:16.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.118+0000 7f9275ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:16.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.118+0000 7f9275ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:16.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.118+0000 7f9275ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:16.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.118+0000 7f9275ffb640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:01:16.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.118+0000 7f9275ffb640 1 -- 172.21.15.112:0/2816943610 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9264004eb0 con 0x7f928013f650 2024-03-29T11:01:16.100 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.119+0000 7f9275ffb640 1 -- 172.21.15.112:0/2816943610 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3438633422 0 0) 0x7f9264004eb0 con 0x7f928013f650 2024-03-29T11:01:16.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.119+0000 7f9275ffb640 1 -- 172.21.15.112:0/2816943610 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9280141300 con 0x7f928013f650 2024-03-29T11:01:16.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.119+0000 7f9275ffb640 1 -- 172.21.15.112:0/2816943610 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f927c0018c0 con 0x7f928013f650 2024-03-29T11:01:16.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.119+0000 7f9275ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:16.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.119+0000 7f9275ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:16.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.119+0000 7f9275ffb640 10 monclient(hunting): dump: 2024-03-29T11:01:16.101 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:16.101 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:16.101 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:16.101 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:16.101 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:16.101 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:16.101 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:16.101 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:16.101 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:16.102 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:16.102 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.119+0000 7f9275ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:16.102 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.122+0000 7f9275ffb640 10 client.?.objecter ms_handle_connect 0x7f92800a9d70 2024-03-29T11:01:16.102 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.122+0000 7f9275ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:16.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.122+0000 7f9275ffb640 1 -- 172.21.15.112:0/2816943610 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3109503686 0 0) 0x7f92801450d0 con 0x7f92800a9d70 2024-03-29T11:01:16.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.122+0000 7f9275ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:16.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.122+0000 7f9275ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:16.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.122+0000 7f9275ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:16.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.122+0000 7f9275ffb640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:01:16.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.122+0000 7f9275ffb640 1 -- 172.21.15.112:0/2816943610 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9264004eb0 con 0x7f92800a9d70 2024-03-29T11:01:16.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.122+0000 7f9275ffb640 1 -- 172.21.15.112:0/2816943610 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2384385835 0 0) 0x7f9280141300 con 0x7f928013f650 2024-03-29T11:01:16.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.123+0000 7f9275ffb640 1 -- 172.21.15.112:0/2816943610 >> v1:172.21.15.112:6790/0 conn(0x7f92800a9d70 legacy=0x7f9280141a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:16.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.123+0000 7f9275ffb640 1 -- 172.21.15.112:0/2816943610 >> v1:172.21.15.112:6789/0 conn(0x7f92801457f0 legacy=0x7f92801ef590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:01:16.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.123+0000 7f9275ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:01:16.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.123+0000 7f9275ffb640 1 monclient: found mon.b 2024-03-29T11:01:16.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.123+0000 7f9275ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:16.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.123+0000 7f9275ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:01:16.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.123+0000 7f9275ffb640 1 -- 172.21.15.112:0/2816943610 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9280145260 con 0x7f928013f650 2024-03-29T11:01:16.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.123+0000 7f9275ffb640 10 monclient: _finish_auth 0 2024-03-29T11:01:16.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.123+0000 7f9275ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:01:16.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.123+0000 7f9275ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:16.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.123+0000 7f92875f4640 5 monclient: authenticate success, global_id 4552 2024-03-29T11:01:16.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.123+0000 7f92875f4640 10 monclient: _renew_subs 2024-03-29T11:01:16.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.123+0000 7f92875f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:01:16.104 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.123+0000 7f92875f4640 1 -- 172.21.15.112:0/2816943610 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9280144c10 con 0x7f928013f650 2024-03-29T11:01:16.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.123+0000 7f92875f4640 10 client.4552.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:16.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.123+0000 7f92875f4640 10 monclient: _renew_subs 2024-03-29T11:01:16.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.123+0000 7f92875f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:01:16.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.123+0000 7f92875f4640 1 -- 172.21.15.112:0/2816943610 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92801422f0 con 0x7f928013f650 2024-03-29T11:01:16.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.123+0000 7f92875f4640 1 librados: init done 2024-03-29T11:01:16.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.125+0000 7f92875f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:16.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.125+0000 7f92875f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:16.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.125+0000 7f92875f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:01:16.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.125+0000 7f92875f4640 1 -- 172.21.15.112:0/2816943610 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92801448d0 con 0x7f928013f650 2024-03-29T11:01:16.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.125+0000 7f9275ffb640 1 -- 172.21.15.112:0/2816943610 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f927c001a20 con 0x7f928013f650 2024-03-29T11:01:16.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.125+0000 7f9275ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:16.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.125+0000 7f9275ffb640 1 -- 172.21.15.112:0/2816943610 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f927c004920 con 0x7f928013f650 2024-03-29T11:01:16.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.125+0000 7f92777fe640 4 set_mon_vals no callback set 2024-03-29T11:01:16.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.125+0000 7f9275ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:16.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.126+0000 7f9275ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:16.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.126+0000 7f9275ffb640 10 monclient: dump: 2024-03-29T11:01:16.107 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:16.107 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:16.107 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:16.107 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:16.107 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:16.107 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:16.107 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:16.107 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:16.107 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:16.108 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:16.109 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.127+0000 7f9275ffb640 1 -- 172.21.15.112:0/2816943610 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f927c01a8f0 con 0x7f928013f650 2024-03-29T11:01:16.109 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.128+0000 7f9275ffb640 1 -- 172.21.15.112:0/2816943610 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3769653263 0 0) 0x7f927c084890 con 0x7f928013f650 2024-03-29T11:01:16.109 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.128+0000 7f9275ffb640 10 client.4552.objecter ms_dispatch 0x7f9280000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:01:16.109 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.128+0000 7f9275ffb640 3 client.4552.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:01:16.109 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.128+0000 7f9275ffb640 3 client.4552.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:01:16.109 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.128+0000 7f9275ffb640 20 client.4552.objecter dump_active .. 0 homeless 2024-03-29T11:01:16.109 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.128+0000 7f9285390640 10 monclient: get_auth_request con 0x7f926406dd70 auth_method 0 2024-03-29T11:01:16.109 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.129+0000 7f9275ffb640 10 client.4552.objecter ms_handle_connect 0x7f926406dd70 2024-03-29T11:01:16.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.133+0000 7f9275ffb640 1 -- 172.21.15.112:0/2816943610 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f927c050170 con 0x7f928013f650 2024-03-29T11:01:16.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.133+0000 7f9275ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:16.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.133+0000 7f9275ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:16.421 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.441+0000 7f92875f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:16.421 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.441+0000 7f92875f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:16.421 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.441+0000 7f92875f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:01:16.421 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.441+0000 7f92875f4640 1 -- 172.21.15.112:0/2816943610 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92800b3680 con 0x7f928013f650 2024-03-29T11:01:16.422 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.441+0000 7f9275ffb640 1 -- 172.21.15.112:0/2816943610 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4236594938) 0x7f927c0520a0 con 0x7f928013f650 2024-03-29T11:01:16.422 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.441+0000 7f9275ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:16.422 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.441+0000 7f9275ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:16.422 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T11:01:16.422 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":11,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T11:01:14.980235+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:01:14.980235+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:16.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.443+0000 7f92875f4640 10 librados: watch_flush enter 2024-03-29T11:01:16.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.444+0000 7f92875f4640 10 librados: watch_flush exit 2024-03-29T11:01:16.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.444+0000 7f92875f4640 20 client.4552.objecter shutdown clearing up homeless session... 2024-03-29T11:01:16.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.444+0000 7f92875f4640 10 client.4552.objecter successfully canceled tick 2024-03-29T11:01:16.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.444+0000 7f92875f4640 1 -- 172.21.15.112:0/2816943610 >> v1:172.21.15.118:6816/3181073408 conn(0x7f926406dd70 legacy=0x7f9264070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:16.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.444+0000 7f92875f4640 10 monclient: shutdown 2024-03-29T11:01:16.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.444+0000 7f92875f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:16.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.444+0000 7f92875f4640 1 -- 172.21.15.112:0/2816943610 >> v1:172.21.15.118:6789/0 conn(0x7f928013f650 legacy=0x7f92801d7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:16.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.444+0000 7f92875f4640 1 -- 172.21.15.112:0/2816943610 shutdown_connections 2024-03-29T11:01:16.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.444+0000 7f92875f4640 1 -- 172.21.15.112:0/2816943610 >> 172.21.15.112:0/2816943610 conn(0x7f9280066a00 msgr2=0x7f928013b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:16.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.444+0000 7f92875f4640 1 -- 172.21.15.112:0/2816943610 shutdown_connections 2024-03-29T11:01:16.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.444+0000 7f92875f4640 1 -- 172.21.15.112:0/2816943610 wait complete. 2024-03-29T11:01:16.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:16.444+0000 7f92875f4640 1 librados: shutdown 2024-03-29T11:01:16.439 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:23.246 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.419+0000 7fb41993d640 10 monclient: get_monmap_and_config 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.419+0000 7fb41993d640 10 monclient: build_initial_monmap 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.419+0000 7fb41993d640 10 monclient: monmap: 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T11:01:23.421379+0000 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T11:01:23.421379+0000 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:23.401 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:01:23.401 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T11:01:23.401 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T11:01:23.401 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:23.401 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.420+0000 7fb41993d640 1 Processor -- start 2024-03-29T11:01:23.402 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.420+0000 7fb41993d640 1 -- start start 2024-03-29T11:01:23.402 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.420+0000 7fb41993d640 10 monclient: init 2024-03-29T11:01:23.402 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.421+0000 7fb41993d640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:23.402 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.421+0000 7fb41993d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:23.402 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.421+0000 7fb41993d640 10 monclient(hunting): picked mon.noname-c con 0x7fb414070170 addr v1:172.21.15.118:6789/0 2024-03-29T11:01:23.402 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.421+0000 7fb41993d640 10 monclient(hunting): picked mon.noname-a con 0x7fb4140713f0 addr v1:172.21.15.112:6789/0 2024-03-29T11:01:23.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.421+0000 7fb41993d640 10 monclient(hunting): picked mon.noname-b con 0x7fb414070550 addr v1:172.21.15.112:6790/0 2024-03-29T11:01:23.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.421+0000 7fb41993d640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb41413dbd0 con 0x7fb4140713f0 2024-03-29T11:01:23.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.421+0000 7fb41993d640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb41413ff70 con 0x7fb414070170 2024-03-29T11:01:23.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.421+0000 7fb41993d640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb414140250 con 0x7fb414070550 2024-03-29T11:01:23.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.421+0000 7fb41993d640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:23.403 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.421+0000 7fb41993d640 10 monclient(hunting): authenticate will time out at 1438.727079s 2024-03-29T11:01:23.406 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.426+0000 7fb4137fe640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fb414070550 0x7fb41414b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:48752/0 (socket says 172.21.15.112:48752) 2024-03-29T11:01:23.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.426+0000 7fb4137fe640 1 -- 172.21.15.112:0/160970440 learned_addr learned my addr 172.21.15.112:0/160970440 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:01:23.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.428+0000 7fb411ffb640 1 -- 172.21.15.112:0/160970440 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199623757 0 0) 0x7fb41413ff70 con 0x7fb414070170 2024-03-29T11:01:23.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.428+0000 7fb411ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:23.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.428+0000 7fb411ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:23.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.428+0000 7fb411ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:23.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.428+0000 7fb411ffb640 10 monclient(hunting): my global_id is 4555 2024-03-29T11:01:23.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.428+0000 7fb411ffb640 1 -- 172.21.15.112:0/160970440 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3fc003520 con 0x7fb414070170 2024-03-29T11:01:23.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.428+0000 7fb411ffb640 1 -- 172.21.15.112:0/160970440 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1041109781 0 0) 0x7fb3fc003520 con 0x7fb414070170 2024-03-29T11:01:23.411 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.428+0000 7fb411ffb640 1 -- 172.21.15.112:0/160970440 >> v1:172.21.15.112:6790/0 conn(0x7fb414070550 legacy=0x7fb41414b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.428+0000 7fb411ffb640 1 -- 172.21.15.112:0/160970440 >> v1:172.21.15.112:6789/0 conn(0x7fb4140713f0 legacy=0x7fb4141406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.428+0000 7fb411ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.428+0000 7fb411ffb640 1 monclient: found mon.noname-c 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.428+0000 7fb411ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.428+0000 7fb411ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.118:6789/0 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.428+0000 7fb411ffb640 1 -- 172.21.15.112:0/160970440 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb414140df0 con 0x7fb414070170 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.428+0000 7fb411ffb640 10 monclient: _finish_auth 0 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.428+0000 7fb411ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.428+0000 7fb411ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.428+0000 7fb411ffb640 1 -- 172.21.15.112:0/160970440 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb408001690 con 0x7fb414070170 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.428+0000 7fb411ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.428+0000 7fb411ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.428+0000 7fb411ffb640 10 monclient: dump: 2024-03-29T11:01:23.412 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:23.413 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:23.413 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:23.413 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:23.413 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:23.413 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:23.413 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:23.413 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:23.413 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:23.413 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:23.413 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.428+0000 7fb41993d640 5 monclient: authenticate success, global_id 4555 2024-03-29T11:01:23.413 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.428+0000 7fb41993d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:23.414 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.430+0000 7fb411ffb640 1 -- 172.21.15.112:0/160970440 <== mon.1 v1:172.21.15.118:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb408001920 con 0x7fb414070170 2024-03-29T11:01:23.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.430+0000 7fb411ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:23.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.430+0000 7fb411ffb640 1 -- 172.21.15.112:0/160970440 <== mon.1 v1:172.21.15.118:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb408001cb0 con 0x7fb414070170 2024-03-29T11:01:23.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.430+0000 7fb411ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:23.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.430+0000 7fb411ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:23.415 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.430+0000 7fb411ffb640 10 monclient: dump: 2024-03-29T11:01:23.415 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:23.415 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:23.415 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:23.415 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:23.415 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:23.415 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:23.415 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:23.415 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:23.416 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:23.416 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:23.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.431+0000 7fb41993d640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:23.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.431+0000 7fb41993d640 4 set_mon_vals no callback set 2024-03-29T11:01:23.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.431+0000 7fb41993d640 10 monclient: shutdown 2024-03-29T11:01:23.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.431+0000 7fb41993d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:23.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.431+0000 7fb41993d640 1 -- 172.21.15.112:0/160970440 >> v1:172.21.15.118:6789/0 conn(0x7fb414070170 legacy=0x7fb41413d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:23.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.432+0000 7fb41993d640 1 -- 172.21.15.112:0/160970440 shutdown_connections 2024-03-29T11:01:23.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.432+0000 7fb41993d640 1 -- 172.21.15.112:0/160970440 >> 172.21.15.112:0/160970440 conn(0x7fb414067610 msgr2=0x7fb4140b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:23.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.432+0000 7fb41993d640 1 -- 172.21.15.112:0/160970440 shutdown_connections 2024-03-29T11:01:23.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.432+0000 7fb41993d640 1 -- 172.21.15.112:0/160970440 wait complete. 2024-03-29T11:01:23.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.433+0000 7fb41993d640 10 monclient: build_initial_monmap 2024-03-29T11:01:23.416 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.433+0000 7fb41993d640 10 monclient: monmap: 2024-03-29T11:01:23.416 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:01:23.416 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:23.417 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T11:01:23.417 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T11:01:23.417 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:23.417 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:23.417 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:01:23.417 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T11:01:23.417 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T11:01:23.417 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:23.417 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.433+0000 7fb41993d640 1 librados: starting msgr at 2024-03-29T11:01:23.417 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.433+0000 7fb41993d640 1 librados: starting objecter 2024-03-29T11:01:23.417 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.433+0000 7fb41993d640 1 Processor -- start 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.434+0000 7fb41993d640 1 -- start start 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.434+0000 7fb41993d640 1 librados: setting wanted keys 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.434+0000 7fb41993d640 1 librados: calling monclient init 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.434+0000 7fb41993d640 10 monclient: init 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.434+0000 7fb41993d640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.434+0000 7fb41993d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.435+0000 7fb41993d640 10 monclient(hunting): picked mon.noname-a con 0x7fb414070170 addr v1:172.21.15.112:6789/0 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.435+0000 7fb41993d640 10 monclient(hunting): picked mon.noname-c con 0x7fb414070550 addr v1:172.21.15.112:6790/0 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.435+0000 7fb41993d640 10 monclient(hunting): picked mon.noname-b con 0x7fb4140713f0 addr v1:172.21.15.118:6789/0 2024-03-29T11:01:23.419 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.435+0000 7fb41993d640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb414132d20 con 0x7fb414070170 2024-03-29T11:01:23.419 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.435+0000 7fb41993d640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4141de940 con 0x7fb4140713f0 2024-03-29T11:01:23.419 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.435+0000 7fb41993d640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb41414a180 con 0x7fb414070550 2024-03-29T11:01:23.419 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.435+0000 7fb41993d640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:23.419 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.435+0000 7fb41993d640 10 monclient(hunting): authenticate will time out at 1438.740760s 2024-03-29T11:01:23.420 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.438+0000 7fb412ffd640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fb414070170 0x7fb4141dee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:56149/0 (socket says 172.21.15.112:56149) 2024-03-29T11:01:23.420 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.438+0000 7fb412ffd640 1 -- 172.21.15.112:0/3583861634 learned_addr learned my addr 172.21.15.112:0/3583861634 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:01:23.420 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.438+0000 7fb3fbfff640 10 client.?.objecter ms_handle_connect 0x7fb414070170 2024-03-29T11:01:23.420 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.438+0000 7fb3fbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:23.420 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.438+0000 7fb3fbfff640 1 -- 172.21.15.112:0/3583861634 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498213417 0 0) 0x7fb414132d20 con 0x7fb414070170 2024-03-29T11:01:23.420 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.438+0000 7fb3fbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:23.420 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.438+0000 7fb3fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:23.421 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.438+0000 7fb3fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:23.421 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.438+0000 7fb3fbfff640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:01:23.421 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.439+0000 7fb3fbfff640 1 -- 172.21.15.112:0/3583861634 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb400004eb0 con 0x7fb414070170 2024-03-29T11:01:23.421 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.439+0000 7fb3fbfff640 10 client.?.objecter ms_handle_connect 0x7fb4140713f0 2024-03-29T11:01:23.421 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.439+0000 7fb3fbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.440+0000 7fb3fbfff640 1 -- 172.21.15.112:0/3583861634 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276343208 0 0) 0x7fb4141de940 con 0x7fb4140713f0 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.440+0000 7fb3fbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.440+0000 7fb3fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.440+0000 7fb3fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.440+0000 7fb3fbfff640 10 monclient(hunting): my global_id is 4558 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.440+0000 7fb3fbfff640 1 -- 172.21.15.112:0/3583861634 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb414132d20 con 0x7fb4140713f0 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.440+0000 7fb3fbfff640 10 client.?.objecter ms_handle_connect 0x7fb414070550 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.440+0000 7fb3fbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.440+0000 7fb3fbfff640 1 -- 172.21.15.112:0/3583861634 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 334293586 0 0) 0x7fb400004eb0 con 0x7fb414070170 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.440+0000 7fb3fbfff640 1 -- 172.21.15.112:0/3583861634 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4141de940 con 0x7fb414070170 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.440+0000 7fb3fbfff640 1 -- 172.21.15.112:0/3583861634 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb408001690 con 0x7fb414070170 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.440+0000 7fb3fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.440+0000 7fb3fbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.440+0000 7fb3fbfff640 10 monclient(hunting): dump: 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.440+0000 7fb3fbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.441+0000 7fb3fbfff640 1 -- 172.21.15.112:0/3583861634 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4260866054 0 0) 0x7fb414132d20 con 0x7fb4140713f0 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.441+0000 7fb3fbfff640 1 -- 172.21.15.112:0/3583861634 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb400004eb0 con 0x7fb4140713f0 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.441+0000 7fb3fbfff640 1 -- 172.21.15.112:0/3583861634 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb40c001890 con 0x7fb4140713f0 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.441+0000 7fb3fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.441+0000 7fb3fbfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.441+0000 7fb3fbfff640 10 monclient(hunting): dump: 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.441+0000 7fb3fbfff640 1 -- 172.21.15.112:0/3583861634 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3606268126 0 0) 0x7fb400004eb0 con 0x7fb4140713f0 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.441+0000 7fb3fbfff640 1 -- 172.21.15.112:0/3583861634 >> v1:172.21.15.112:6790/0 conn(0x7fb414070550 legacy=0x7fb41414ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.441+0000 7fb3fbfff640 1 -- 172.21.15.112:0/3583861634 >> v1:172.21.15.112:6789/0 conn(0x7fb414070170 legacy=0x7fb4141dee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.441+0000 7fb3fbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.441+0000 7fb3fbfff640 1 monclient: found mon.b 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.441+0000 7fb3fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.441+0000 7fb3fbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.441+0000 7fb3fbfff640 1 -- 172.21.15.112:0/3583861634 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41414a310 con 0x7fb4140713f0 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.441+0000 7fb3fbfff640 10 monclient: _finish_auth 0 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.441+0000 7fb3fbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.441+0000 7fb3fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.442+0000 7fb41993d640 5 monclient: authenticate success, global_id 4558 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.442+0000 7fb41993d640 10 monclient: _renew_subs 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.442+0000 7fb41993d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.442+0000 7fb41993d640 1 -- 172.21.15.112:0/3583861634 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb41406bd40 con 0x7fb4140713f0 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.443+0000 7fb41993d640 10 client.4558.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.443+0000 7fb41993d640 10 monclient: _renew_subs 2024-03-29T11:01:23.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.443+0000 7fb41993d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:01:23.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.443+0000 7fb41993d640 1 -- 172.21.15.112:0/3583861634 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4141df7e0 con 0x7fb4140713f0 2024-03-29T11:01:23.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.443+0000 7fb41993d640 1 librados: init done 2024-03-29T11:01:23.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.443+0000 7fb3fbfff640 1 -- 172.21.15.112:0/3583861634 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb40c0019f0 con 0x7fb4140713f0 2024-03-29T11:01:23.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.443+0000 7fb3fbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:23.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.443+0000 7fb3fbfff640 1 -- 172.21.15.112:0/3583861634 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb40c001f40 con 0x7fb4140713f0 2024-03-29T11:01:23.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.443+0000 7fb3fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:23.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.443+0000 7fb3fbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:23.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.443+0000 7fb3fbfff640 10 monclient: dump: 2024-03-29T11:01:23.427 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:23.427 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:23.427 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:23.427 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:23.427 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:23.428 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:23.428 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:23.428 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:23.428 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:23.428 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:23.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.444+0000 7fb4117fa640 4 set_mon_vals no callback set 2024-03-29T11:01:23.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.445+0000 7fb41993d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:23.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.446+0000 7fb41993d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:23.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.446+0000 7fb41993d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:01:23.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.446+0000 7fb41993d640 1 -- 172.21.15.112:0/3583861634 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f4005080 con 0x7fb4140713f0 2024-03-29T11:01:23.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.446+0000 7fb3fbfff640 1 -- 172.21.15.112:0/3583861634 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fb40c01ac80 con 0x7fb4140713f0 2024-03-29T11:01:23.429 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.448+0000 7fb3fbfff640 1 -- 172.21.15.112:0/3583861634 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3769653263 0 0) 0x7fb40c084b20 con 0x7fb4140713f0 2024-03-29T11:01:23.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.448+0000 7fb3fbfff640 10 client.4558.objecter ms_dispatch 0x7fb414000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:01:23.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.448+0000 7fb3fbfff640 3 client.4558.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:01:23.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.448+0000 7fb3fbfff640 3 client.4558.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:01:23.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.448+0000 7fb3fbfff640 20 client.4558.objecter dump_active .. 0 homeless 2024-03-29T11:01:23.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.449+0000 7fb412ffd640 10 monclient: get_auth_request con 0x7fb40006dfe0 auth_method 0 2024-03-29T11:01:23.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.449+0000 7fb3fbfff640 10 client.4558.objecter ms_handle_connect 0x7fb40006dfe0 2024-03-29T11:01:23.438 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.455+0000 7fb3fbfff640 1 -- 172.21.15.112:0/3583861634 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb40c050400 con 0x7fb4140713f0 2024-03-29T11:01:23.438 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.455+0000 7fb3fbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:23.438 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.455+0000 7fb3fbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:23.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.780+0000 7fb41993d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:23.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.780+0000 7fb41993d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:23.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.780+0000 7fb41993d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:01:23.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.780+0000 7fb41993d640 1 -- 172.21.15.112:0/3583861634 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3f4005330 con 0x7fb4140713f0 2024-03-29T11:01:23.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.781+0000 7fb3fbfff640 1 -- 172.21.15.112:0/3583861634 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4236594938) 0x7fb40c052330 con 0x7fb4140713f0 2024-03-29T11:01:23.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.781+0000 7fb3fbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:23.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.781+0000 7fb3fbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:23.763 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T11:01:23.763 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":11,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T11:01:14.980235+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:01:14.980235+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:23.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.785+0000 7fb41993d640 10 librados: watch_flush enter 2024-03-29T11:01:23.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.786+0000 7fb41993d640 10 librados: watch_flush exit 2024-03-29T11:01:23.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.786+0000 7fb41993d640 20 client.4558.objecter shutdown clearing up homeless session... 2024-03-29T11:01:23.767 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.786+0000 7fb41993d640 10 client.4558.objecter successfully canceled tick 2024-03-29T11:01:23.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.787+0000 7fb41993d640 1 -- 172.21.15.112:0/3583861634 >> v1:172.21.15.118:6816/3181073408 conn(0x7fb40006dfe0 legacy=0x7fb4000704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:23.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.787+0000 7fb41993d640 10 monclient: shutdown 2024-03-29T11:01:23.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.787+0000 7fb41993d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:23.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.787+0000 7fb41993d640 1 -- 172.21.15.112:0/3583861634 >> v1:172.21.15.118:6789/0 conn(0x7fb4140713f0 legacy=0x7fb4141e5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:23.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.787+0000 7fb41993d640 1 -- 172.21.15.112:0/3583861634 shutdown_connections 2024-03-29T11:01:23.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.787+0000 7fb41993d640 1 -- 172.21.15.112:0/3583861634 >> 172.21.15.112:0/3583861634 conn(0x7fb414067610 msgr2=0x7fb41413d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:23.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.788+0000 7fb41993d640 1 -- 172.21.15.112:0/3583861634 shutdown_connections 2024-03-29T11:01:23.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.788+0000 7fb41993d640 1 -- 172.21.15.112:0/3583861634 wait complete. 2024-03-29T11:01:23.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:23.788+0000 7fb41993d640 1 librados: shutdown 2024-03-29T11:01:23.788 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:30.594 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:30.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.764+0000 7fb374dad640 10 monclient: get_monmap_and_config 2024-03-29T11:01:30.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.764+0000 7fb374dad640 10 monclient: build_initial_monmap 2024-03-29T11:01:30.744 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.764+0000 7fb374dad640 10 monclient: monmap: 2024-03-29T11:01:30.744 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:01:30.745 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:30.745 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T11:01:30.765439+0000 2024-03-29T11:01:30.745 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T11:01:30.765439+0000 2024-03-29T11:01:30.745 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:30.745 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:30.745 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:01:30.745 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T11:01:30.745 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T11:01:30.745 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:30.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.766+0000 7fb374dad640 1 Processor -- start 2024-03-29T11:01:30.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.766+0000 7fb374dad640 1 -- start start 2024-03-29T11:01:30.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.766+0000 7fb374dad640 10 monclient: init 2024-03-29T11:01:30.746 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.766+0000 7fb374dad640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:30.747 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.766+0000 7fb374dad640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:01:30.747 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.766+0000 7fb374dad640 10 monclient(hunting): picked mon.noname-b con 0x7fb368006890 addr v1:172.21.15.112:6790/0 2024-03-29T11:01:30.747 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.766+0000 7fb374dad640 10 monclient(hunting): picked mon.noname-c con 0x7fb368006c70 addr v1:172.21.15.118:6789/0 2024-03-29T11:01:30.747 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.767+0000 7fb36f7fe640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fb368006890 0x7fb3680c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:38044/0 (socket says 172.21.15.112:38044) 2024-03-29T11:01:30.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.769+0000 7fb36f7fe640 1 -- 172.21.15.112:0/1895494634 learned_addr learned my addr 172.21.15.112:0/1895494634 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:01:30.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.769+0000 7fb374dad640 10 monclient(hunting): picked mon.noname-a con 0x7fb36802b4c0 addr v1:172.21.15.112:6789/0 2024-03-29T11:01:30.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.769+0000 7fb374dad640 1 -- 172.21.15.112:0/1895494634 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36802c830 con 0x7fb36802b4c0 2024-03-29T11:01:30.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.770+0000 7fb374dad640 1 -- 172.21.15.112:0/1895494634 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36802afa0 con 0x7fb368006c70 2024-03-29T11:01:30.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.770+0000 7fb374dad640 1 -- 172.21.15.112:0/1895494634 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36802aba0 con 0x7fb368006890 2024-03-29T11:01:30.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.770+0000 7fb374dad640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:30.750 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.770+0000 7fb374dad640 10 monclient(hunting): authenticate will time out at 1446.074424s 2024-03-29T11:01:30.754 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.770+0000 7fb36e7fc640 1 -- 172.21.15.112:0/1895494634 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960674855 0 0) 0x7fb36802aba0 con 0x7fb368006890 2024-03-29T11:01:30.754 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.770+0000 7fb36e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:01:30.754 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.770+0000 7fb36e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:30.754 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.770+0000 7fb36e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:30.754 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.770+0000 7fb36e7fc640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:01:30.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.770+0000 7fb36e7fc640 1 -- 172.21.15.112:0/1895494634 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb360003520 con 0x7fb368006890 2024-03-29T11:01:30.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 1 -- 172.21.15.112:0/1895494634 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718717193 0 0) 0x7fb36802c830 con 0x7fb36802b4c0 2024-03-29T11:01:30.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:01:30.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:30.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:30.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:01:30.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 1 -- 172.21.15.112:0/1895494634 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb36802aba0 con 0x7fb36802b4c0 2024-03-29T11:01:30.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 1 -- 172.21.15.112:0/1895494634 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4173441818 0 0) 0x7fb360003520 con 0x7fb368006890 2024-03-29T11:01:30.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 1 -- 172.21.15.112:0/1895494634 >> v1:172.21.15.118:6789/0 conn(0x7fb368006c70 legacy=0x7fb3680c6d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:30.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 1 -- 172.21.15.112:0/1895494634 >> v1:172.21.15.112:6789/0 conn(0x7fb36802b4c0 legacy=0x7fb3680cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:30.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:01:30.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:01:30.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:30.755 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T11:01:30.756 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 1 -- 172.21.15.112:0/1895494634 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36802ad30 con 0x7fb368006890 2024-03-29T11:01:30.756 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:01:30.756 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:01:30.756 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:30.756 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 1 -- 172.21.15.112:0/1895494634 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb364001690 con 0x7fb368006890 2024-03-29T11:01:30.756 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:30.756 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:30.756 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 10 monclient: dump: 2024-03-29T11:01:30.756 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:30.756 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:30.756 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:30.756 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:30.756 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:30.756 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 1 -- 172.21.15.112:0/1895494634 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3640017f0 con 0x7fb368006890 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 1 -- 172.21.15.112:0/1895494634 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb364001c40 con 0x7fb368006890 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.771+0000 7fb36e7fc640 10 monclient: dump: 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:30.758 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:30.758 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:30.758 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:30.758 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:30.758 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:30.758 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:30.759 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:30.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.772+0000 7fb374dad640 5 monclient: authenticate success, global_id 4640 2024-03-29T11:01:30.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.772+0000 7fb374dad640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:30.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.772+0000 7fb374dad640 4 set_mon_vals no callback set 2024-03-29T11:01:30.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.773+0000 7fb374dad640 10 monclient: shutdown 2024-03-29T11:01:30.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.773+0000 7fb374dad640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:30.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.773+0000 7fb374dad640 1 -- 172.21.15.112:0/1895494634 >> v1:172.21.15.112:6790/0 conn(0x7fb368006890 legacy=0x7fb3680c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:30.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.773+0000 7fb374dad640 1 -- 172.21.15.112:0/1895494634 shutdown_connections 2024-03-29T11:01:30.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.773+0000 7fb374dad640 1 -- 172.21.15.112:0/1895494634 >> 172.21.15.112:0/1895494634 conn(0x7fb3680058f0 msgr2=0x7fb368005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:30.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.774+0000 7fb374dad640 1 -- 172.21.15.112:0/1895494634 shutdown_connections 2024-03-29T11:01:30.759 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.774+0000 7fb374dad640 1 -- 172.21.15.112:0/1895494634 wait complete. 2024-03-29T11:01:30.761 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.775+0000 7fb374dad640 10 monclient: build_initial_monmap 2024-03-29T11:01:30.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.775+0000 7fb374dad640 10 monclient: monmap: 2024-03-29T11:01:30.762 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:01:30.762 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:30.762 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T11:01:30.762 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T11:01:30.762 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:30.762 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:30.762 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:01:30.762 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T11:01:30.762 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T11:01:30.762 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:30.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.775+0000 7fb374dad640 1 librados: starting msgr at 2024-03-29T11:01:30.762 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.775+0000 7fb374dad640 1 librados: starting objecter 2024-03-29T11:01:30.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.775+0000 7fb374dad640 1 Processor -- start 2024-03-29T11:01:30.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.776+0000 7fb374dad640 1 -- start start 2024-03-29T11:01:30.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.776+0000 7fb374dad640 1 librados: setting wanted keys 2024-03-29T11:01:30.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.776+0000 7fb374dad640 1 librados: calling monclient init 2024-03-29T11:01:30.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.776+0000 7fb374dad640 10 monclient: init 2024-03-29T11:01:30.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.777+0000 7fb374dad640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:30.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.777+0000 7fb374dad640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:01:30.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.777+0000 7fb374dad640 10 monclient(hunting): picked mon.noname-c con 0x7fb368006890 addr v1:172.21.15.112:6790/0 2024-03-29T11:01:30.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.778+0000 7fb374dad640 10 monclient(hunting): picked mon.noname-b con 0x7fb36802b4c0 addr v1:172.21.15.118:6789/0 2024-03-29T11:01:30.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.778+0000 7fb374dad640 10 monclient(hunting): picked mon.noname-a con 0x7fb36802cd10 addr v1:172.21.15.112:6789/0 2024-03-29T11:01:30.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.778+0000 7fb374dad640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3680c3630 con 0x7fb36802cd10 2024-03-29T11:01:30.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.778+0000 7fb374dad640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3680c3910 con 0x7fb36802b4c0 2024-03-29T11:01:30.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.779+0000 7fb374dad640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3680c15c0 con 0x7fb368006890 2024-03-29T11:01:30.763 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.779+0000 7fb374dad640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:30.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.779+0000 7fb374dad640 10 monclient(hunting): authenticate will time out at 1446.083210s 2024-03-29T11:01:30.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.779+0000 7fb36f7fe640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fb368006890 0x7fb3680c3ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:42148/0 (socket says 172.21.15.112:42148) 2024-03-29T11:01:30.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.779+0000 7fb36f7fe640 1 -- 172.21.15.112:0/281998842 learned_addr learned my addr 172.21.15.112:0/281998842 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:01:30.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.779+0000 7fb35bfff640 10 client.?.objecter ms_handle_connect 0x7fb368006890 2024-03-29T11:01:30.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.779+0000 7fb35bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:30.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.779+0000 7fb35bfff640 1 -- 172.21.15.112:0/281998842 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025847000 0 0) 0x7fb3680c15c0 con 0x7fb368006890 2024-03-29T11:01:30.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.779+0000 7fb35bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:30.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.779+0000 7fb35bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:30.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.779+0000 7fb35bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:30.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.779+0000 7fb35bfff640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:01:30.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.779+0000 7fb35bfff640 1 -- 172.21.15.112:0/281998842 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb354004eb0 con 0x7fb368006890 2024-03-29T11:01:30.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.780+0000 7fb35bfff640 1 -- 172.21.15.112:0/281998842 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3197129815 0 0) 0x7fb354004eb0 con 0x7fb368006890 2024-03-29T11:01:30.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.780+0000 7fb35bfff640 1 -- 172.21.15.112:0/281998842 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3680c15c0 con 0x7fb368006890 2024-03-29T11:01:30.764 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.780+0000 7fb35bfff640 1 -- 172.21.15.112:0/281998842 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb364001bc0 con 0x7fb368006890 2024-03-29T11:01:30.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.780+0000 7fb35bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:30.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.780+0000 7fb35bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:30.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.780+0000 7fb35bfff640 10 monclient(hunting): dump: 2024-03-29T11:01:30.765 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:30.765 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:30.765 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:30.765 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:30.765 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:30.765 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:30.765 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:30.765 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:30.765 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:30.765 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:30.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.780+0000 7fb35bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:30.765 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.780+0000 7fb35bfff640 1 -- 172.21.15.112:0/281998842 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3156802134 0 0) 0x7fb3680c15c0 con 0x7fb368006890 2024-03-29T11:01:30.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.780+0000 7fb35bfff640 1 -- 172.21.15.112:0/281998842 >> v1:172.21.15.118:6789/0 conn(0x7fb36802b4c0 legacy=0x7fb3680bbd90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:01:30.766 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.780+0000 7fb35bfff640 1 -- 172.21.15.112:0/281998842 >> v1:172.21.15.112:6789/0 conn(0x7fb36802cd10 legacy=0x7fb368163ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:01:30.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.784+0000 7fb35bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:01:30.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.784+0000 7fb35bfff640 1 monclient: found mon.c 2024-03-29T11:01:30.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.784+0000 7fb35bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:30.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.784+0000 7fb35bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:01:30.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.784+0000 7fb35bfff640 1 -- 172.21.15.112:0/281998842 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3680c45b0 con 0x7fb368006890 2024-03-29T11:01:30.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.784+0000 7fb35bfff640 10 monclient: _finish_auth 0 2024-03-29T11:01:30.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.784+0000 7fb35bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:01:30.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.784+0000 7fb35bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:30.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.784+0000 7fb374dad640 5 monclient: authenticate success, global_id 4643 2024-03-29T11:01:30.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.784+0000 7fb374dad640 10 monclient: _renew_subs 2024-03-29T11:01:30.768 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.784+0000 7fb374dad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:01:30.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.784+0000 7fb374dad640 1 -- 172.21.15.112:0/281998842 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3680c4820 con 0x7fb368006890 2024-03-29T11:01:30.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.784+0000 7fb374dad640 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:30.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.784+0000 7fb374dad640 10 monclient: _renew_subs 2024-03-29T11:01:30.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.784+0000 7fb374dad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:01:30.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.784+0000 7fb374dad640 1 -- 172.21.15.112:0/281998842 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb368031c20 con 0x7fb368006890 2024-03-29T11:01:30.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.784+0000 7fb374dad640 1 librados: init done 2024-03-29T11:01:30.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.787+0000 7fb35bfff640 1 -- 172.21.15.112:0/281998842 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb364004b00 con 0x7fb368006890 2024-03-29T11:01:30.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.787+0000 7fb35bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:30.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.787+0000 7fb374dad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:30.769 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.787+0000 7fb35bfff640 1 -- 172.21.15.112:0/281998842 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fb364004f50 con 0x7fb368006890 2024-03-29T11:01:30.770 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.787+0000 7fb35bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:30.770 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.787+0000 7fb36dffb640 4 set_mon_vals no callback set 2024-03-29T11:01:30.770 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.787+0000 7fb35bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:30.770 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.787+0000 7fb35bfff640 10 monclient: dump: 2024-03-29T11:01:30.770 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:30.770 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:30.770 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:30.770 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:30.770 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:30.770 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:30.770 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:30.770 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:30.770 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:30.770 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:30.771 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.788+0000 7fb374dad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:30.771 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.788+0000 7fb374dad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:01:30.771 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.788+0000 7fb374dad640 1 -- 172.21.15.112:0/281998842 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3680c3b50 con 0x7fb368006890 2024-03-29T11:01:30.771 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.789+0000 7fb35bfff640 1 -- 172.21.15.112:0/281998842 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fb3640050b0 con 0x7fb368006890 2024-03-29T11:01:30.771 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.790+0000 7fb35bfff640 1 -- 172.21.15.112:0/281998842 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3769653263 0 0) 0x7fb364084bf0 con 0x7fb368006890 2024-03-29T11:01:30.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.790+0000 7fb35bfff640 10 client.4643.objecter ms_dispatch 0x7fb370000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:01:30.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.790+0000 7fb35bfff640 3 client.4643.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:01:30.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.790+0000 7fb35bfff640 3 client.4643.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:01:30.772 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.790+0000 7fb35bfff640 20 client.4643.objecter dump_active .. 0 homeless 2024-03-29T11:01:30.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.793+0000 7fb36effd640 10 monclient: get_auth_request con 0x7fb35406dc00 auth_method 0 2024-03-29T11:01:30.773 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.794+0000 7fb35bfff640 10 client.4643.objecter ms_handle_connect 0x7fb35406dc00 2024-03-29T11:01:30.778 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.798+0000 7fb35bfff640 1 -- 172.21.15.112:0/281998842 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb3640504d0 con 0x7fb368006890 2024-03-29T11:01:30.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.798+0000 7fb35bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:30.779 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:30.798+0000 7fb35bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:31.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:31.127+0000 7fb374dad640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:31.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:31.127+0000 7fb374dad640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:31.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:31.127+0000 7fb374dad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:01:31.107 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:31.127+0000 7fb374dad640 1 -- 172.21.15.112:0/281998842 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb36802dd70 con 0x7fb368006890 2024-03-29T11:01:31.109 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T11:01:31.109 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":11,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T11:01:14.980235+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:01:14.980235+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:31.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:31.128+0000 7fb35bfff640 1 -- 172.21.15.112:0/281998842 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4236594938) 0x7fb364052400 con 0x7fb368006890 2024-03-29T11:01:31.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:31.128+0000 7fb35bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:31.110 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:31.128+0000 7fb35bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:31.111 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:31.131+0000 7fb374dad640 10 librados: watch_flush enter 2024-03-29T11:01:31.111 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:31.132+0000 7fb374dad640 10 librados: watch_flush exit 2024-03-29T11:01:31.111 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:31.132+0000 7fb374dad640 20 client.4643.objecter shutdown clearing up homeless session... 2024-03-29T11:01:31.111 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:31.132+0000 7fb374dad640 10 client.4643.objecter successfully canceled tick 2024-03-29T11:01:31.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:31.133+0000 7fb374dad640 1 -- 172.21.15.112:0/281998842 >> v1:172.21.15.118:6816/3181073408 conn(0x7fb35406dc00 legacy=0x7fb3540700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:31.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:31.133+0000 7fb374dad640 10 monclient: shutdown 2024-03-29T11:01:31.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:31.133+0000 7fb374dad640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:31.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:31.133+0000 7fb374dad640 1 -- 172.21.15.112:0/281998842 >> v1:172.21.15.112:6790/0 conn(0x7fb368006890 legacy=0x7fb3680c3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:31.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:31.133+0000 7fb374dad640 1 -- 172.21.15.112:0/281998842 shutdown_connections 2024-03-29T11:01:31.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:31.133+0000 7fb374dad640 1 -- 172.21.15.112:0/281998842 >> 172.21.15.112:0/281998842 conn(0x7fb3680058f0 msgr2=0x7fb368038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:31.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:31.133+0000 7fb374dad640 1 -- 172.21.15.112:0/281998842 shutdown_connections 2024-03-29T11:01:31.114 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:31.133+0000 7fb374dad640 1 -- 172.21.15.112:0/281998842 wait complete. 2024-03-29T11:01:31.114 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:31.134+0000 7fb374dad640 1 librados: shutdown 2024-03-29T11:01:31.130 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:37.937 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:38.085 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.105+0000 7f121883d640 10 monclient: get_monmap_and_config 2024-03-29T11:01:38.085 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.105+0000 7f121883d640 10 monclient: build_initial_monmap 2024-03-29T11:01:38.085 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.105+0000 7f121883d640 10 monclient: monmap: 2024-03-29T11:01:38.086 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:01:38.086 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:38.086 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T11:01:38.106803+0000 2024-03-29T11:01:38.086 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T11:01:38.106803+0000 2024-03-29T11:01:38.086 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:38.086 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:38.086 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:01:38.086 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T11:01:38.086 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T11:01:38.086 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:38.087 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.107+0000 7f121883d640 1 Processor -- start 2024-03-29T11:01:38.087 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.107+0000 7f121883d640 1 -- start start 2024-03-29T11:01:38.087 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.107+0000 7f121883d640 10 monclient: init 2024-03-29T11:01:38.087 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.108+0000 7f121883d640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:38.088 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.108+0000 7f121883d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:38.088 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.108+0000 7f121883d640 10 monclient(hunting): picked mon.noname-c con 0x7f120c006e30 addr v1:172.21.15.118:6789/0 2024-03-29T11:01:38.088 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.108+0000 7f121883d640 10 monclient(hunting): picked mon.noname-a con 0x7f120c035e00 addr v1:172.21.15.112:6789/0 2024-03-29T11:01:38.088 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.108+0000 7f121883d640 10 monclient(hunting): picked mon.noname-b con 0x7f120c0c9670 addr v1:172.21.15.112:6790/0 2024-03-29T11:01:38.088 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.108+0000 7f121883d640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120c035970 con 0x7f120c035e00 2024-03-29T11:01:38.088 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.108+0000 7f121883d640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120c036280 con 0x7f120c006e30 2024-03-29T11:01:38.088 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.108+0000 7f121883d640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120c035640 con 0x7f120c0c9670 2024-03-29T11:01:38.088 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.108+0000 7f121883d640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:38.088 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.108+0000 7f121883d640 10 monclient(hunting): authenticate will time out at 1453.412480s 2024-03-29T11:01:38.089 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.109+0000 7f121359e640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f120c0c9670 0x7f120c0d7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:53912/0 (socket says 172.21.15.112:53912) 2024-03-29T11:01:38.089 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.109+0000 7f121359e640 1 -- 172.21.15.112:0/951190741 learned_addr learned my addr 172.21.15.112:0/951190741 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.111+0000 7f1211d9b640 1 -- 172.21.15.112:0/951190741 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693645497 0 0) 0x7f120c035640 con 0x7f120c0c9670 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.111+0000 7f1211d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.111+0000 7f1211d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.111+0000 7f1211d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.111+0000 7f1211d9b640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.111+0000 7f1211d9b640 1 -- 172.21.15.112:0/951190741 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11fc003520 con 0x7f120c0c9670 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.111+0000 7f1211d9b640 1 -- 172.21.15.112:0/951190741 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2069580512 0 0) 0x7f11fc003520 con 0x7f120c0c9670 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.111+0000 7f1211d9b640 1 -- 172.21.15.112:0/951190741 >> v1:172.21.15.118:6789/0 conn(0x7f120c006e30 legacy=0x7f120c0cbcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.111+0000 7f1211d9b640 1 -- 172.21.15.112:0/951190741 >> v1:172.21.15.112:6789/0 conn(0x7f120c035e00 legacy=0x7f120c0c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.111+0000 7f1211d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.111+0000 7f1211d9b640 1 monclient: found mon.noname-b 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.111+0000 7f1211d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.111+0000 7f1211d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.111+0000 7f1211d9b640 1 -- 172.21.15.112:0/951190741 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120c034e20 con 0x7f120c0c9670 2024-03-29T11:01:38.093 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.111+0000 7f1211d9b640 10 monclient: _finish_auth 0 2024-03-29T11:01:38.093 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.111+0000 7f1211d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:01:38.093 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.111+0000 7f1211d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:38.093 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.111+0000 7f1211d9b640 1 -- 172.21.15.112:0/951190741 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1204001690 con 0x7f120c0c9670 2024-03-29T11:01:38.093 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.111+0000 7f1211d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:38.093 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.111+0000 7f1211d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:38.093 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.111+0000 7f1211d9b640 10 monclient: dump: 2024-03-29T11:01:38.093 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:38.093 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:38.093 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:38.093 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:38.093 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:38.093 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:38.093 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:38.094 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:38.094 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:38.094 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:38.094 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.111+0000 7f121883d640 5 monclient: authenticate success, global_id 4646 2024-03-29T11:01:38.094 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.111+0000 7f121883d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:38.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.112+0000 7f1211d9b640 1 -- 172.21.15.112:0/951190741 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12040018f0 con 0x7f120c0c9670 2024-03-29T11:01:38.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.112+0000 7f1211d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:38.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.112+0000 7f1211d9b640 1 -- 172.21.15.112:0/951190741 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1204001d40 con 0x7f120c0c9670 2024-03-29T11:01:38.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.112+0000 7f1211d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:38.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.112+0000 7f1211d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:38.096 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.112+0000 7f1211d9b640 10 monclient: dump: 2024-03-29T11:01:38.096 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:38.096 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:38.096 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:38.096 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:38.096 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:38.097 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:38.097 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:38.097 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:38.097 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:38.097 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:38.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.113+0000 7f121883d640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:38.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.113+0000 7f121883d640 4 set_mon_vals no callback set 2024-03-29T11:01:38.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.114+0000 7f121883d640 10 monclient: shutdown 2024-03-29T11:01:38.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.114+0000 7f121883d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:38.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.114+0000 7f121883d640 1 -- 172.21.15.112:0/951190741 >> v1:172.21.15.112:6790/0 conn(0x7f120c0c9670 legacy=0x7f120c0d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:38.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.115+0000 7f121883d640 1 -- 172.21.15.112:0/951190741 shutdown_connections 2024-03-29T11:01:38.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.115+0000 7f121883d640 1 -- 172.21.15.112:0/951190741 >> 172.21.15.112:0/951190741 conn(0x7f120c005c00 msgr2=0x7f120c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:38.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.115+0000 7f121883d640 1 -- 172.21.15.112:0/951190741 shutdown_connections 2024-03-29T11:01:38.097 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.116+0000 7f121883d640 1 -- 172.21.15.112:0/951190741 wait complete. 2024-03-29T11:01:38.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.119+0000 7f121883d640 10 monclient: build_initial_monmap 2024-03-29T11:01:38.099 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.120+0000 7f121883d640 10 monclient: monmap: 2024-03-29T11:01:38.099 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:01:38.099 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:38.099 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T11:01:38.100 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T11:01:38.100 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:38.100 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:38.100 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:01:38.100 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T11:01:38.100 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T11:01:38.100 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:38.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.120+0000 7f121883d640 1 librados: starting msgr at 2024-03-29T11:01:38.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.120+0000 7f121883d640 1 librados: starting objecter 2024-03-29T11:01:38.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.121+0000 7f121883d640 1 Processor -- start 2024-03-29T11:01:38.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.121+0000 7f121883d640 1 -- start start 2024-03-29T11:01:38.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.121+0000 7f121883d640 1 librados: setting wanted keys 2024-03-29T11:01:38.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.121+0000 7f121883d640 1 librados: calling monclient init 2024-03-29T11:01:38.101 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.121+0000 7f121883d640 10 monclient: init 2024-03-29T11:01:38.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.122+0000 7f121883d640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:38.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.122+0000 7f121883d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:01:38.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.122+0000 7f1212d9d640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f120c006e30 0x7f120c033830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:58774/0 (socket says 172.21.15.112:58774) 2024-03-29T11:01:38.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.122+0000 7f1212d9d640 1 -- 172.21.15.112:0/3841001010 learned_addr learned my addr 172.21.15.112:0/3841001010 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:01:38.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.122+0000 7f121883d640 10 monclient(hunting): picked mon.noname-c con 0x7f120c006e30 addr v1:172.21.15.112:6790/0 2024-03-29T11:01:38.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.122+0000 7f121883d640 10 monclient(hunting): picked mon.noname-b con 0x7f120c035e00 addr v1:172.21.15.118:6789/0 2024-03-29T11:01:38.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.122+0000 7f11fb7fe640 10 client.?.objecter ms_handle_connect 0x7f120c006e30 2024-03-29T11:01:38.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.122+0000 7f11fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:38.103 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.124+0000 7f121883d640 10 monclient(hunting): picked mon.noname-a con 0x7f120c0c9670 addr v1:172.21.15.112:6789/0 2024-03-29T11:01:38.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.124+0000 7f121883d640 1 -- 172.21.15.112:0/3841001010 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120c0333a0 con 0x7f120c0c9670 2024-03-29T11:01:38.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.124+0000 7f121883d640 1 -- 172.21.15.112:0/3841001010 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120c033010 con 0x7f120c035e00 2024-03-29T11:01:38.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.124+0000 7f121883d640 1 -- 172.21.15.112:0/3841001010 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120c0cdbe0 con 0x7f120c006e30 2024-03-29T11:01:38.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.124+0000 7f121883d640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:38.106 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.124+0000 7f121883d640 10 monclient(hunting): authenticate will time out at 1453.428338s 2024-03-29T11:01:38.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.127+0000 7f11fb7fe640 10 client.?.objecter ms_handle_connect 0x7f120c035e00 2024-03-29T11:01:38.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.127+0000 7f11fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:38.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.127+0000 7f11fb7fe640 10 client.?.objecter ms_handle_connect 0x7f120c0c9670 2024-03-29T11:01:38.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.127+0000 7f11fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:38.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.127+0000 7f11fb7fe640 1 -- 172.21.15.112:0/3841001010 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862119928 0 0) 0x7f120c0cdbe0 con 0x7f120c006e30 2024-03-29T11:01:38.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.127+0000 7f11fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:38.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.127+0000 7f11fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:38.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.128+0000 7f11fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:38.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.128+0000 7f11fb7fe640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:01:38.113 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.128+0000 7f11fb7fe640 1 -- 172.21.15.112:0/3841001010 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1200004eb0 con 0x7f120c006e30 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.128+0000 7f11fb7fe640 1 -- 172.21.15.112:0/3841001010 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 413147098 0 0) 0x7f120c033010 con 0x7f120c035e00 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.128+0000 7f11fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.128+0000 7f11fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.128+0000 7f11fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.128+0000 7f11fb7fe640 10 monclient(hunting): my global_id is 4561 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.128+0000 7f11fb7fe640 1 -- 172.21.15.112:0/3841001010 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f120c0cdbe0 con 0x7f120c035e00 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.128+0000 7f11fb7fe640 1 -- 172.21.15.112:0/3841001010 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1363245520 0 0) 0x7f120c0333a0 con 0x7f120c0c9670 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.128+0000 7f11fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.128+0000 7f11fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.128+0000 7f11fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.129+0000 7f11fb7fe640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.129+0000 7f11fb7fe640 1 -- 172.21.15.112:0/3841001010 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f120c033010 con 0x7f120c0c9670 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.129+0000 7f11fb7fe640 1 -- 172.21.15.112:0/3841001010 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 472655484 0 0) 0x7f120c0cdbe0 con 0x7f120c035e00 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.129+0000 7f11fb7fe640 1 -- 172.21.15.112:0/3841001010 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f120c0333a0 con 0x7f120c035e00 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.129+0000 7f11fb7fe640 1 -- 172.21.15.112:0/3841001010 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f121404fdd0 con 0x7f120c035e00 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.129+0000 7f11fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.129+0000 7f11fb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.129+0000 7f11fb7fe640 10 monclient(hunting): dump: 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:38.116 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:38.116 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:38.116 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:38.116 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:38.116 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:38.116 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.129+0000 7f11fb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:38.116 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.129+0000 7f11fb7fe640 1 -- 172.21.15.112:0/3841001010 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4268659172 0 0) 0x7f120c0333a0 con 0x7f120c035e00 2024-03-29T11:01:38.116 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.129+0000 7f11fb7fe640 1 -- 172.21.15.112:0/3841001010 >> v1:172.21.15.112:6790/0 conn(0x7f120c006e30 legacy=0x7f120c033830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:38.116 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.129+0000 7f11fb7fe640 1 -- 172.21.15.112:0/3841001010 >> v1:172.21.15.112:6789/0 conn(0x7f120c0c9670 legacy=0x7f120c16bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:38.116 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.129+0000 7f11fb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:01:38.116 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.129+0000 7f11fb7fe640 1 monclient: found mon.b 2024-03-29T11:01:38.116 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.129+0000 7f11fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:38.116 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.129+0000 7f11fb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:01:38.116 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.130+0000 7f11fb7fe640 1 -- 172.21.15.112:0/3841001010 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120c0cb0f0 con 0x7f120c035e00 2024-03-29T11:01:38.116 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.130+0000 7f11fb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:01:38.117 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.130+0000 7f11fb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:01:38.117 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.130+0000 7f11fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:38.117 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.130+0000 7f11fb7fe640 1 -- 172.21.15.112:0/3841001010 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f121404f400 con 0x7f120c035e00 2024-03-29T11:01:38.117 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.130+0000 7f11fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:38.117 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.130+0000 7f11fb7fe640 1 -- 172.21.15.112:0/3841001010 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f1214075f10 con 0x7f120c035e00 2024-03-29T11:01:38.117 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.130+0000 7f1210d99640 4 set_mon_vals no callback set 2024-03-29T11:01:38.117 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.130+0000 7f121883d640 5 monclient: authenticate success, global_id 4561 2024-03-29T11:01:38.117 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.130+0000 7f121883d640 10 monclient: _renew_subs 2024-03-29T11:01:38.117 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.130+0000 7f121883d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:01:38.117 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.130+0000 7f121883d640 1 -- 172.21.15.112:0/3841001010 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f120c0cb2e0 con 0x7f120c035e00 2024-03-29T11:01:38.117 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.130+0000 7f121883d640 10 client.4561.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:38.117 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.130+0000 7f121883d640 10 monclient: _renew_subs 2024-03-29T11:01:38.117 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.130+0000 7f121883d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:01:38.117 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.130+0000 7f121883d640 1 -- 172.21.15.112:0/3841001010 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f120c0311d0 con 0x7f120c035e00 2024-03-29T11:01:38.118 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.131+0000 7f121883d640 1 librados: init done 2024-03-29T11:01:38.118 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.131+0000 7f11fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:38.118 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.131+0000 7f11fb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:38.118 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.131+0000 7f11fb7fe640 10 monclient: dump: 2024-03-29T11:01:38.118 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:38.118 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:38.118 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:38.118 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:38.118 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:38.118 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:38.118 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:38.118 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:38.118 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:38.119 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:38.119 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.132+0000 7f121883d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:38.119 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.132+0000 7f121883d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:38.119 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.132+0000 7f121883d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:01:38.119 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.132+0000 7f121883d640 1 -- 172.21.15.112:0/3841001010 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f120c007b90 con 0x7f120c035e00 2024-03-29T11:01:38.119 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.133+0000 7f11fb7fe640 1 -- 172.21.15.112:0/3841001010 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f1214067bf0 con 0x7f120c035e00 2024-03-29T11:01:38.120 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.133+0000 7f1212d9d640 10 monclient: get_auth_request con 0x7f120006dd30 auth_method 0 2024-03-29T11:01:38.120 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.134+0000 7f11fb7fe640 1 -- 172.21.15.112:0/3841001010 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3769653263 0 0) 0x7f12140fe100 con 0x7f120c035e00 2024-03-29T11:01:38.120 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.134+0000 7f11fb7fe640 10 client.4561.objecter ms_dispatch 0x7f1214000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:01:38.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.134+0000 7f11fb7fe640 3 client.4561.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:01:38.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.134+0000 7f11fb7fe640 3 client.4561.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:01:38.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.134+0000 7f11fb7fe640 20 client.4561.objecter dump_active .. 0 homeless 2024-03-29T11:01:38.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.134+0000 7f11fb7fe640 10 client.4561.objecter ms_handle_connect 0x7f120006dd30 2024-03-29T11:01:38.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.140+0000 7f11fb7fe640 1 -- 172.21.15.112:0/3841001010 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f12140c99e0 con 0x7f120c035e00 2024-03-29T11:01:38.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.141+0000 7f11fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:38.121 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.141+0000 7f11fb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:38.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.465+0000 7f121883d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:38.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.465+0000 7f121883d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:38.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.465+0000 7f121883d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:01:38.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.465+0000 7f121883d640 1 -- 172.21.15.112:0/3841001010 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f120c005120 con 0x7f120c035e00 2024-03-29T11:01:38.446 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.467+0000 7f11fb7fe640 1 -- 172.21.15.112:0/3841001010 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4236594938) 0x7f12140cb910 con 0x7f120c035e00 2024-03-29T11:01:38.446 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.467+0000 7f11fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:38.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.467+0000 7f11fb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:38.449 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T11:01:38.450 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":11,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T11:01:14.980235+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:01:14.980235+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:38.451 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.470+0000 7f121883d640 10 librados: watch_flush enter 2024-03-29T11:01:38.451 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.470+0000 7f121883d640 10 librados: watch_flush exit 2024-03-29T11:01:38.451 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.470+0000 7f121883d640 20 client.4561.objecter shutdown clearing up homeless session... 2024-03-29T11:01:38.451 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.470+0000 7f121883d640 10 client.4561.objecter successfully canceled tick 2024-03-29T11:01:38.451 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.471+0000 7f121883d640 1 -- 172.21.15.112:0/3841001010 >> v1:172.21.15.118:6816/3181073408 conn(0x7f120006dd30 legacy=0x7f12000701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:38.451 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.471+0000 7f121883d640 10 monclient: shutdown 2024-03-29T11:01:38.451 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.471+0000 7f121883d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:38.452 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.471+0000 7f121883d640 1 -- 172.21.15.112:0/3841001010 >> v1:172.21.15.118:6789/0 conn(0x7f120c035e00 legacy=0x7f120c032900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:38.452 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.471+0000 7f121883d640 1 -- 172.21.15.112:0/3841001010 shutdown_connections 2024-03-29T11:01:38.452 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.471+0000 7f121883d640 1 -- 172.21.15.112:0/3841001010 >> 172.21.15.112:0/3841001010 conn(0x7f120c005c00 msgr2=0x7f120c0c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:38.452 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.471+0000 7f121883d640 1 -- 172.21.15.112:0/3841001010 shutdown_connections 2024-03-29T11:01:38.452 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.471+0000 7f121883d640 1 -- 172.21.15.112:0/3841001010 wait complete. 2024-03-29T11:01:38.452 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:38.471+0000 7f121883d640 1 librados: shutdown 2024-03-29T11:01:38.468 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:45.274 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:45.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.438+0000 7ff39326b640 10 monclient: get_monmap_and_config 2024-03-29T11:01:45.418 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.438+0000 7ff39326b640 10 monclient: build_initial_monmap 2024-03-29T11:01:45.419 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.438+0000 7ff39326b640 10 monclient: monmap: 2024-03-29T11:01:45.419 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:01:45.419 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:45.419 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T11:01:45.439728+0000 2024-03-29T11:01:45.419 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T11:01:45.439728+0000 2024-03-29T11:01:45.419 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:45.419 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:45.419 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:01:45.419 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T11:01:45.419 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T11:01:45.419 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:45.423 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.439+0000 7ff39326b640 1 Processor -- start 2024-03-29T11:01:45.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.439+0000 7ff39326b640 1 -- start start 2024-03-29T11:01:45.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.439+0000 7ff39326b640 10 monclient: init 2024-03-29T11:01:45.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.440+0000 7ff39326b640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:45.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.440+0000 7ff39326b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:01:45.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.440+0000 7ff39326b640 10 monclient(hunting): picked mon.noname-b con 0x7ff38c1457f0 addr v1:172.21.15.112:6790/0 2024-03-29T11:01:45.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.440+0000 7ff39326b640 10 monclient(hunting): picked mon.noname-c con 0x7ff38c13f650 addr v1:172.21.15.118:6789/0 2024-03-29T11:01:45.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.440+0000 7ff39326b640 10 monclient(hunting): picked mon.noname-a con 0x7ff38c0a9d70 addr v1:172.21.15.112:6789/0 2024-03-29T11:01:45.424 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.440+0000 7ff39326b640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff38c0aaef0 con 0x7ff38c0a9d70 2024-03-29T11:01:45.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.441+0000 7ff391808640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7ff38c0a9d70 0x7ff38c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35006/0 (socket says 172.21.15.112:35006) 2024-03-29T11:01:45.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.441+0000 7ff391808640 1 -- 172.21.15.112:0/2589899518 learned_addr learned my addr 172.21.15.112:0/2589899518 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:01:45.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.441+0000 7ff39326b640 1 -- 172.21.15.112:0/2589899518 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff38c145bd0 con 0x7ff38c13f650 2024-03-29T11:01:45.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.441+0000 7ff39326b640 1 -- 172.21.15.112:0/2589899518 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff38c145eb0 con 0x7ff38c1457f0 2024-03-29T11:01:45.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.441+0000 7ff39326b640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:45.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.441+0000 7ff39326b640 10 monclient(hunting): authenticate will time out at 1460.745332s 2024-03-29T11:01:45.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.441+0000 7ff383fff640 1 -- 172.21.15.112:0/2589899518 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243559983 0 0) 0x7ff38c0aaef0 con 0x7ff38c0a9d70 2024-03-29T11:01:45.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.441+0000 7ff383fff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:45.425 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.441+0000 7ff383fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:45.426 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.441+0000 7ff383fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:45.426 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.441+0000 7ff383fff640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:01:45.426 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.441+0000 7ff383fff640 1 -- 172.21.15.112:0/2589899518 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff378003520 con 0x7ff38c0a9d70 2024-03-29T11:01:45.426 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.441+0000 7ff383fff640 1 -- 172.21.15.112:0/2589899518 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3332790616 0 0) 0x7ff378003520 con 0x7ff38c0a9d70 2024-03-29T11:01:45.426 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.442+0000 7ff383fff640 1 -- 172.21.15.112:0/2589899518 >> v1:172.21.15.112:6790/0 conn(0x7ff38c1457f0 legacy=0x7ff38c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:01:45.426 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.442+0000 7ff383fff640 1 -- 172.21.15.112:0/2589899518 >> v1:172.21.15.118:6789/0 conn(0x7ff38c13f650 legacy=0x7ff38c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:45.426 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.442+0000 7ff383fff640 10 monclient: _finish_hunting 0 2024-03-29T11:01:45.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.442+0000 7ff383fff640 1 monclient: found mon.noname-a 2024-03-29T11:01:45.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.442+0000 7ff383fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:45.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.442+0000 7ff383fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T11:01:45.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.442+0000 7ff383fff640 1 -- 172.21.15.112:0/2589899518 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38c146340 con 0x7ff38c0a9d70 2024-03-29T11:01:45.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.442+0000 7ff383fff640 10 monclient: _finish_auth 0 2024-03-29T11:01:45.427 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.442+0000 7ff383fff640 10 monclient: _check_auth_tickets 2024-03-29T11:01:45.428 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.442+0000 7ff383fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:45.428 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.442+0000 7ff383fff640 1 -- 172.21.15.112:0/2589899518 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7ff384001a10 con 0x7ff38c0a9d70 2024-03-29T11:01:45.428 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.442+0000 7ff383fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:45.428 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.442+0000 7ff383fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:45.428 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.442+0000 7ff383fff640 10 monclient: dump: 2024-03-29T11:01:45.428 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:45.428 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:45.429 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:45.429 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:45.429 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:45.429 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:45.429 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:45.429 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:45.429 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:45.429 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:45.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.442+0000 7ff39326b640 5 monclient: authenticate success, global_id 4569 2024-03-29T11:01:45.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.442+0000 7ff39326b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:45.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.442+0000 7ff383fff640 1 -- 172.21.15.112:0/2589899518 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff384001be0 con 0x7ff38c0a9d70 2024-03-29T11:01:45.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.442+0000 7ff383fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:45.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.442+0000 7ff383fff640 1 -- 172.21.15.112:0/2589899518 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7ff384001ff0 con 0x7ff38c0a9d70 2024-03-29T11:01:45.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.442+0000 7ff383fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:45.430 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.442+0000 7ff383fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:45.431 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.442+0000 7ff383fff640 10 monclient: dump: 2024-03-29T11:01:45.431 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:45.431 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:45.431 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:45.431 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:45.431 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:45.431 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:45.432 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:45.432 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:45.432 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:45.432 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:45.432 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.442+0000 7ff39326b640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:45.432 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.442+0000 7ff39326b640 4 set_mon_vals no callback set 2024-03-29T11:01:45.433 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.443+0000 7ff39326b640 10 monclient: shutdown 2024-03-29T11:01:45.433 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.443+0000 7ff39326b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:45.433 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.443+0000 7ff39326b640 1 -- 172.21.15.112:0/2589899518 >> v1:172.21.15.112:6789/0 conn(0x7ff38c0a9d70 legacy=0x7ff38c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:45.433 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.443+0000 7ff39326b640 1 -- 172.21.15.112:0/2589899518 shutdown_connections 2024-03-29T11:01:45.433 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.443+0000 7ff39326b640 1 -- 172.21.15.112:0/2589899518 >> 172.21.15.112:0/2589899518 conn(0x7ff38c066a00 msgr2=0x7ff38c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:45.433 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.443+0000 7ff39326b640 1 -- 172.21.15.112:0/2589899518 shutdown_connections 2024-03-29T11:01:45.434 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.443+0000 7ff39326b640 1 -- 172.21.15.112:0/2589899518 wait complete. 2024-03-29T11:01:45.439 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.445+0000 7ff39326b640 10 monclient: build_initial_monmap 2024-03-29T11:01:45.440 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.445+0000 7ff39326b640 10 monclient: monmap: 2024-03-29T11:01:45.440 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:01:45.440 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:45.440 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T11:01:45.440 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T11:01:45.440 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:45.440 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:45.440 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:01:45.440 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T11:01:45.440 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T11:01:45.440 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:45.440 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.445+0000 7ff39326b640 1 librados: starting msgr at 2024-03-29T11:01:45.441 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.445+0000 7ff39326b640 1 librados: starting objecter 2024-03-29T11:01:45.441 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.445+0000 7ff39326b640 1 Processor -- start 2024-03-29T11:01:45.441 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.445+0000 7ff39326b640 1 -- start start 2024-03-29T11:01:45.441 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.445+0000 7ff39326b640 1 librados: setting wanted keys 2024-03-29T11:01:45.441 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.446+0000 7ff39326b640 1 librados: calling monclient init 2024-03-29T11:01:45.441 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.446+0000 7ff39326b640 10 monclient: init 2024-03-29T11:01:45.441 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.447+0000 7ff39326b640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:45.441 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.447+0000 7ff39326b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:45.441 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.447+0000 7ff39326b640 10 monclient(hunting): picked mon.noname-b con 0x7ff38c0a9d70 addr v1:172.21.15.118:6789/0 2024-03-29T11:01:45.441 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.447+0000 7ff39326b640 10 monclient(hunting): picked mon.noname-a con 0x7ff38c13f650 addr v1:172.21.15.112:6789/0 2024-03-29T11:01:45.441 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.447+0000 7ff391007640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7ff38c0a9d70 0x7ff38c1df0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.112:46480/0 (socket says 172.21.15.112:46480) 2024-03-29T11:01:45.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.447+0000 7ff39326b640 10 monclient(hunting): picked mon.noname-c con 0x7ff38c1457f0 addr v1:172.21.15.112:6790/0 2024-03-29T11:01:45.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.447+0000 7ff391007640 1 -- 172.21.15.112:0/2518304138 learned_addr learned my addr 172.21.15.112:0/2518304138 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:01:45.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.447+0000 7ff39326b640 1 -- 172.21.15.112:0/2518304138 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff38c0720c0 con 0x7ff38c13f650 2024-03-29T11:01:45.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.447+0000 7ff39326b640 1 -- 172.21.15.112:0/2518304138 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff38c071600 con 0x7ff38c0a9d70 2024-03-29T11:01:45.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.447+0000 7ff39326b640 1 -- 172.21.15.112:0/2518304138 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff38c070590 con 0x7ff38c1457f0 2024-03-29T11:01:45.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.447+0000 7ff39326b640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:45.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.447+0000 7ff39326b640 10 monclient(hunting): authenticate will time out at 1460.751756s 2024-03-29T11:01:45.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.447+0000 7ff381ffb640 10 client.?.objecter ms_handle_connect 0x7ff38c0a9d70 2024-03-29T11:01:45.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.447+0000 7ff381ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:45.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.448+0000 7ff381ffb640 10 client.?.objecter ms_handle_connect 0x7ff38c13f650 2024-03-29T11:01:45.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.448+0000 7ff381ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:45.442 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.448+0000 7ff381ffb640 10 client.?.objecter ms_handle_connect 0x7ff38c1457f0 2024-03-29T11:01:45.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.448+0000 7ff381ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:45.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.448+0000 7ff381ffb640 1 -- 172.21.15.112:0/2518304138 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583920111 0 0) 0x7ff38c0720c0 con 0x7ff38c13f650 2024-03-29T11:01:45.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.448+0000 7ff381ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:45.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.448+0000 7ff381ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:45.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.448+0000 7ff381ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:45.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.448+0000 7ff381ffb640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:01:45.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.448+0000 7ff381ffb640 1 -- 172.21.15.112:0/2518304138 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff370004eb0 con 0x7ff38c13f650 2024-03-29T11:01:45.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.448+0000 7ff381ffb640 1 -- 172.21.15.112:0/2518304138 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2964160809 0 0) 0x7ff38c070590 con 0x7ff38c1457f0 2024-03-29T11:01:45.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.448+0000 7ff381ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:45.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.448+0000 7ff381ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:45.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.448+0000 7ff381ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:45.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.448+0000 7ff381ffb640 10 monclient(hunting): my global_id is 4652 2024-03-29T11:01:45.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.448+0000 7ff381ffb640 1 -- 172.21.15.112:0/2518304138 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff38c0720c0 con 0x7ff38c1457f0 2024-03-29T11:01:45.443 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.448+0000 7ff381ffb640 1 -- 172.21.15.112:0/2518304138 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2041576218 0 0) 0x7ff38c071600 con 0x7ff38c0a9d70 2024-03-29T11:01:45.444 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.448+0000 7ff381ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:45.444 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.448+0000 7ff381ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:45.444 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.448+0000 7ff381ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:45.444 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.448+0000 7ff381ffb640 10 monclient(hunting): my global_id is 4567 2024-03-29T11:01:45.444 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.448+0000 7ff381ffb640 1 -- 172.21.15.112:0/2518304138 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff38c070590 con 0x7ff38c0a9d70 2024-03-29T11:01:45.444 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff381ffb640 1 -- 172.21.15.112:0/2518304138 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3141320754 0 0) 0x7ff370004eb0 con 0x7ff38c13f650 2024-03-29T11:01:45.444 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff381ffb640 1 -- 172.21.15.112:0/2518304138 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff38c071600 con 0x7ff38c13f650 2024-03-29T11:01:45.444 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff381ffb640 1 -- 172.21.15.112:0/2518304138 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7ff37c001930 con 0x7ff38c13f650 2024-03-29T11:01:45.444 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff381ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:45.444 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff381ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:45.444 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff381ffb640 10 monclient(hunting): dump: 2024-03-29T11:01:45.444 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:45.444 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:45.444 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:45.444 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:45.445 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:45.445 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:45.445 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:45.445 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:45.445 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:45.445 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:45.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff381ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:45.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff381ffb640 1 -- 172.21.15.112:0/2518304138 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2754455782 0 0) 0x7ff38c071600 con 0x7ff38c13f650 2024-03-29T11:01:45.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff381ffb640 1 -- 172.21.15.112:0/2518304138 >> v1:172.21.15.112:6790/0 conn(0x7ff38c1457f0 legacy=0x7ff38c0b0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:45.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff381ffb640 1 -- 172.21.15.112:0/2518304138 >> v1:172.21.15.118:6789/0 conn(0x7ff38c0a9d70 legacy=0x7ff38c1df0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:45.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff381ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:01:45.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff381ffb640 1 monclient: found mon.a 2024-03-29T11:01:45.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff381ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:45.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff381ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T11:01:45.445 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff381ffb640 1 -- 172.21.15.112:0/2518304138 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38c1df8f0 con 0x7ff38c13f650 2024-03-29T11:01:45.446 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff381ffb640 10 monclient: _finish_auth 0 2024-03-29T11:01:45.446 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff381ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:01:45.446 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff381ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:45.446 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff39326b640 5 monclient: authenticate success, global_id 4572 2024-03-29T11:01:45.446 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff39326b640 10 monclient: _renew_subs 2024-03-29T11:01:45.446 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff39326b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T11:01:45.446 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff39326b640 1 -- 172.21.15.112:0/2518304138 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff38c0b0d60 con 0x7ff38c13f650 2024-03-29T11:01:45.446 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff39326b640 10 client.4572.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:45.446 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff39326b640 10 monclient: _renew_subs 2024-03-29T11:01:45.446 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff39326b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T11:01:45.446 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff39326b640 1 -- 172.21.15.112:0/2518304138 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff38c06f110 con 0x7ff38c13f650 2024-03-29T11:01:45.446 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.449+0000 7ff39326b640 1 librados: init done 2024-03-29T11:01:45.446 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.450+0000 7ff381ffb640 1 -- 172.21.15.112:0/2518304138 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff37c001a90 con 0x7ff38c13f650 2024-03-29T11:01:45.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.450+0000 7ff381ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:45.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.451+0000 7ff39326b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:45.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.451+0000 7ff381ffb640 1 -- 172.21.15.112:0/2518304138 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7ff37c003830 con 0x7ff38c13f650 2024-03-29T11:01:45.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.451+0000 7ff381ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:45.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.451+0000 7ff381ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:45.447 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.451+0000 7ff381ffb640 10 monclient: dump: 2024-03-29T11:01:45.447 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:45.447 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:45.447 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:45.447 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:45.447 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:45.447 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:45.447 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:45.447 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:45.448 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:45.448 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:45.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.451+0000 7ff382ffd640 4 set_mon_vals no callback set 2024-03-29T11:01:45.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.452+0000 7ff381ffb640 1 -- 172.21.15.112:0/2518304138 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7ff37c003a50 con 0x7ff38c13f650 2024-03-29T11:01:45.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.452+0000 7ff381ffb640 1 -- 172.21.15.112:0/2518304138 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3769653263 0 0) 0x7ff37c0837d0 con 0x7ff38c13f650 2024-03-29T11:01:45.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.452+0000 7ff381ffb640 10 client.4572.objecter ms_dispatch 0x7ff38c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:01:45.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.452+0000 7ff381ffb640 3 client.4572.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:01:45.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.452+0000 7ff381ffb640 3 client.4572.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:01:45.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.452+0000 7ff381ffb640 20 client.4572.objecter dump_active .. 0 homeless 2024-03-29T11:01:45.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.452+0000 7ff39326b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:45.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.452+0000 7ff39326b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T11:01:45.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.452+0000 7ff39326b640 1 -- 172.21.15.112:0/2518304138 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff38c075f10 con 0x7ff38c13f650 2024-03-29T11:01:45.448 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.452+0000 7ff391007640 10 monclient: get_auth_request con 0x7ff37006dc00 auth_method 0 2024-03-29T11:01:45.449 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.453+0000 7ff381ffb640 10 client.4572.objecter ms_handle_connect 0x7ff37006dc00 2024-03-29T11:01:45.449 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.462+0000 7ff381ffb640 1 -- 172.21.15.112:0/2518304138 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff37c04f1f0 con 0x7ff38c13f650 2024-03-29T11:01:45.449 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.462+0000 7ff381ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:45.449 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.462+0000 7ff381ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:45.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.821+0000 7ff39326b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:45.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.821+0000 7ff39326b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:45.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.821+0000 7ff39326b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T11:01:45.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.821+0000 7ff39326b640 1 -- 172.21.15.112:0/2518304138 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff38c13bb60 con 0x7ff38c13f650 2024-03-29T11:01:45.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.822+0000 7ff381ffb640 1 -- 172.21.15.112:0/2518304138 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4236594938) 0x7ff37c051120 con 0x7ff38c13f650 2024-03-29T11:01:45.802 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.822+0000 7ff381ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:45.803 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T11:01:45.803 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":11,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T11:01:14.980235+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:01:14.980235+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:45.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.822+0000 7ff381ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:45.806 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.825+0000 7ff39326b640 10 librados: watch_flush enter 2024-03-29T11:01:45.806 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.825+0000 7ff39326b640 10 librados: watch_flush exit 2024-03-29T11:01:45.806 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.825+0000 7ff39326b640 20 client.4572.objecter shutdown clearing up homeless session... 2024-03-29T11:01:45.807 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.825+0000 7ff39326b640 10 client.4572.objecter successfully canceled tick 2024-03-29T11:01:45.807 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.825+0000 7ff39326b640 1 -- 172.21.15.112:0/2518304138 >> v1:172.21.15.118:6816/3181073408 conn(0x7ff37006dc00 legacy=0x7ff3700700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:45.807 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.825+0000 7ff39326b640 10 monclient: shutdown 2024-03-29T11:01:45.807 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.825+0000 7ff39326b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:45.807 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.825+0000 7ff39326b640 1 -- 172.21.15.112:0/2518304138 >> v1:172.21.15.112:6789/0 conn(0x7ff38c13f650 legacy=0x7ff38c0719b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:45.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.825+0000 7ff39326b640 1 -- 172.21.15.112:0/2518304138 shutdown_connections 2024-03-29T11:01:45.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.826+0000 7ff39326b640 1 -- 172.21.15.112:0/2518304138 >> 172.21.15.112:0/2518304138 conn(0x7ff38c066a00 msgr2=0x7ff38c0b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:45.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.826+0000 7ff39326b640 1 -- 172.21.15.112:0/2518304138 shutdown_connections 2024-03-29T11:01:45.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.826+0000 7ff39326b640 1 -- 172.21.15.112:0/2518304138 wait complete. 2024-03-29T11:01:45.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:45.826+0000 7ff39326b640 1 librados: shutdown 2024-03-29T11:01:45.820 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:52.627 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:52.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.814+0000 7ff61d071640 10 monclient: get_monmap_and_config 2024-03-29T11:01:52.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.814+0000 7ff61d071640 10 monclient: build_initial_monmap 2024-03-29T11:01:52.794 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.814+0000 7ff61d071640 10 monclient: monmap: 2024-03-29T11:01:52.794 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:01:52.794 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:52.794 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T11:01:52.815098+0000 2024-03-29T11:01:52.794 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T11:01:52.815098+0000 2024-03-29T11:01:52.794 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:52.794 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:52.795 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:01:52.795 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T11:01:52.795 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T11:01:52.795 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:52.795 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.815+0000 7ff61d071640 1 Processor -- start 2024-03-29T11:01:52.795 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.815+0000 7ff61d071640 1 -- start start 2024-03-29T11:01:52.795 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.815+0000 7ff61d071640 10 monclient: init 2024-03-29T11:01:52.796 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.816+0000 7ff61d071640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:52.796 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.816+0000 7ff61d071640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:01:52.796 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.816+0000 7ff61d071640 10 monclient(hunting): picked mon.noname-c con 0x7ff6100389e0 addr v1:172.21.15.118:6789/0 2024-03-29T11:01:52.796 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.816+0000 7ff61d071640 10 monclient(hunting): picked mon.noname-b con 0x7ff610038dc0 addr v1:172.21.15.112:6790/0 2024-03-29T11:01:52.796 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.816+0000 7ff61d071640 10 monclient(hunting): picked mon.noname-a con 0x7ff610030660 addr v1:172.21.15.112:6789/0 2024-03-29T11:01:52.796 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.816+0000 7ff61d071640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6100c54c0 con 0x7ff610030660 2024-03-29T11:01:52.796 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.816+0000 7ff61d071640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff610036430 con 0x7ff6100389e0 2024-03-29T11:01:52.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.816+0000 7ff61d071640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff610039840 con 0x7ff610038dc0 2024-03-29T11:01:52.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.816+0000 7ff61d071640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:52.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.816+0000 7ff61d071640 10 monclient(hunting): authenticate will time out at 1468.120778s 2024-03-29T11:01:52.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.816+0000 7ff6177fe640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7ff610038dc0 0x7ff6100cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:36366/0 (socket says 172.21.15.112:36366) 2024-03-29T11:01:52.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.816+0000 7ff6177fe640 1 -- 172.21.15.112:0/3240474201 learned_addr learned my addr 172.21.15.112:0/3240474201 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:01:52.797 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.817+0000 7ff616ffd640 1 -- 172.21.15.112:0/3240474201 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2305233456 0 0) 0x7ff610039840 con 0x7ff610038dc0 2024-03-29T11:01:52.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.817+0000 7ff616ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:52.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.817+0000 7ff616ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:52.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.817+0000 7ff616ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:52.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.817+0000 7ff616ffd640 10 monclient(hunting): my global_id is 4655 2024-03-29T11:01:52.798 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.818+0000 7ff616ffd640 1 -- 172.21.15.112:0/3240474201 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff608003520 con 0x7ff610038dc0 2024-03-29T11:01:52.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.819+0000 7ff616ffd640 1 -- 172.21.15.112:0/3240474201 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 517429363 0 0) 0x7ff608003520 con 0x7ff610038dc0 2024-03-29T11:01:52.801 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.819+0000 7ff616ffd640 1 -- 172.21.15.112:0/3240474201 >> v1:172.21.15.118:6789/0 conn(0x7ff6100389e0 legacy=0x7ff610033520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:52.803 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.823+0000 7ff616ffd640 1 -- 172.21.15.112:0/3240474201 >> v1:172.21.15.112:6789/0 conn(0x7ff610030660 legacy=0x7ff6100d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:52.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.823+0000 7ff616ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:01:52.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.823+0000 7ff616ffd640 1 monclient: found mon.noname-b 2024-03-29T11:01:52.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.823+0000 7ff616ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:52.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.823+0000 7ff616ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T11:01:52.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.823+0000 7ff616ffd640 1 -- 172.21.15.112:0/3240474201 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff610035690 con 0x7ff610038dc0 2024-03-29T11:01:52.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.823+0000 7ff616ffd640 10 monclient: _finish_auth 0 2024-03-29T11:01:52.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.823+0000 7ff616ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:01:52.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.823+0000 7ff616ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:52.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.823+0000 7ff616ffd640 1 -- 172.21.15.112:0/3240474201 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7ff604001690 con 0x7ff610038dc0 2024-03-29T11:01:52.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.823+0000 7ff616ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:52.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.823+0000 7ff616ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:52.804 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.823+0000 7ff616ffd640 10 monclient: dump: 2024-03-29T11:01:52.804 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:52.804 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:52.805 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:52.805 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:52.805 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:52.805 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:52.805 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:52.805 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:52.805 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:52.805 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:52.805 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.823+0000 7ff61d071640 5 monclient: authenticate success, global_id 4655 2024-03-29T11:01:52.805 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.823+0000 7ff61d071640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:52.807 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.828+0000 7ff616ffd640 1 -- 172.21.15.112:0/3240474201 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff604001860 con 0x7ff610038dc0 2024-03-29T11:01:52.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.828+0000 7ff616ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:52.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.828+0000 7ff61d071640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:52.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.828+0000 7ff61d071640 4 set_mon_vals no callback set 2024-03-29T11:01:52.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.828+0000 7ff616ffd640 1 -- 172.21.15.112:0/3240474201 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7ff604001c70 con 0x7ff610038dc0 2024-03-29T11:01:52.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.828+0000 7ff616ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:52.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.828+0000 7ff616ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:52.808 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.828+0000 7ff616ffd640 10 monclient: dump: 2024-03-29T11:01:52.808 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:52.808 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:52.808 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:52.808 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:52.808 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:52.808 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:52.809 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:52.809 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:52.809 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:52.809 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:52.809 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.829+0000 7ff61d071640 10 monclient: shutdown 2024-03-29T11:01:52.809 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.829+0000 7ff61d071640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:52.809 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.829+0000 7ff61d071640 1 -- 172.21.15.112:0/3240474201 >> v1:172.21.15.112:6790/0 conn(0x7ff610038dc0 legacy=0x7ff6100cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:52.811 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.829+0000 7ff61d071640 1 -- 172.21.15.112:0/3240474201 shutdown_connections 2024-03-29T11:01:52.811 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.829+0000 7ff61d071640 1 -- 172.21.15.112:0/3240474201 >> 172.21.15.112:0/3240474201 conn(0x7ff610005c00 msgr2=0x7ff610006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:52.812 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.833+0000 7ff61d071640 1 -- 172.21.15.112:0/3240474201 shutdown_connections 2024-03-29T11:01:52.813 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.833+0000 7ff61d071640 1 -- 172.21.15.112:0/3240474201 wait complete. 2024-03-29T11:01:52.814 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.834+0000 7ff61d071640 10 monclient: build_initial_monmap 2024-03-29T11:01:52.815 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.834+0000 7ff61d071640 10 monclient: monmap: 2024-03-29T11:01:52.815 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:01:52.815 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:52.815 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T11:01:52.815 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T11:01:52.815 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:52.815 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:52.815 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:01:52.815 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T11:01:52.815 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T11:01:52.815 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:52.815 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.835+0000 7ff61d071640 1 librados: starting msgr at 2024-03-29T11:01:52.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.835+0000 7ff61d071640 1 librados: starting objecter 2024-03-29T11:01:52.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.835+0000 7ff61d071640 1 Processor -- start 2024-03-29T11:01:52.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.835+0000 7ff61d071640 1 -- start start 2024-03-29T11:01:52.816 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.835+0000 7ff61d071640 1 librados: setting wanted keys 2024-03-29T11:01:52.817 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.835+0000 7ff61d071640 1 librados: calling monclient init 2024-03-29T11:01:52.817 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.835+0000 7ff61d071640 10 monclient: init 2024-03-29T11:01:52.817 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.836+0000 7ff61d071640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:52.817 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.836+0000 7ff61d071640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:01:52.817 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.836+0000 7ff61d071640 10 monclient(hunting): picked mon.noname-a con 0x7ff610030660 addr v1:172.21.15.112:6789/0 2024-03-29T11:01:52.818 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.838+0000 7ff61d071640 10 monclient(hunting): picked mon.noname-b con 0x7ff6100389e0 addr v1:172.21.15.118:6789/0 2024-03-29T11:01:52.818 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.838+0000 7ff61d071640 10 monclient(hunting): picked mon.noname-c con 0x7ff6100c7b30 addr v1:172.21.15.112:6790/0 2024-03-29T11:01:52.818 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.838+0000 7ff61d071640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6100c2d00 con 0x7ff610030660 2024-03-29T11:01:52.818 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.838+0000 7ff61d071640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6100c7fa0 con 0x7ff6100389e0 2024-03-29T11:01:52.818 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.838+0000 7ff61d071640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6100c5d60 con 0x7ff6100c7b30 2024-03-29T11:01:52.818 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.838+0000 7ff61d071640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:52.818 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.838+0000 7ff61d071640 10 monclient(hunting): authenticate will time out at 1468.142319s 2024-03-29T11:01:52.819 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.839+0000 7ff61c870640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7ff6100c7b30 0x7ff6100c5650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:38596/0 (socket says 172.21.15.112:38596) 2024-03-29T11:01:52.819 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.839+0000 7ff61c870640 1 -- 172.21.15.112:0/2105968081 learned_addr learned my addr 172.21.15.112:0/2105968081 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:01:52.819 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.839+0000 7ff614ff9640 10 client.?.objecter ms_handle_connect 0x7ff6100c7b30 2024-03-29T11:01:52.819 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.839+0000 7ff614ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:52.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.840+0000 7ff614ff9640 1 -- 172.21.15.112:0/2105968081 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3133210961 0 0) 0x7ff6100c5d60 con 0x7ff6100c7b30 2024-03-29T11:01:52.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.840+0000 7ff614ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:01:52.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.840+0000 7ff614ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:52.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.840+0000 7ff614ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:52.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.840+0000 7ff614ff9640 10 monclient(hunting): my global_id is 4658 2024-03-29T11:01:52.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.840+0000 7ff614ff9640 1 -- 172.21.15.112:0/2105968081 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5fc004eb0 con 0x7ff6100c7b30 2024-03-29T11:01:52.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.840+0000 7ff614ff9640 10 client.?.objecter ms_handle_connect 0x7ff6100389e0 2024-03-29T11:01:52.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.840+0000 7ff614ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:52.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.840+0000 7ff614ff9640 1 -- 172.21.15.112:0/2105968081 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3765939631 0 0) 0x7ff6100c7fa0 con 0x7ff6100389e0 2024-03-29T11:01:52.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.840+0000 7ff614ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:01:52.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.840+0000 7ff614ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:52.820 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.840+0000 7ff614ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:52.821 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.840+0000 7ff614ff9640 10 monclient(hunting): my global_id is 4570 2024-03-29T11:01:52.821 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.840+0000 7ff614ff9640 1 -- 172.21.15.112:0/2105968081 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6100c5d60 con 0x7ff6100389e0 2024-03-29T11:01:52.822 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.843+0000 7ff614ff9640 1 -- 172.21.15.112:0/2105968081 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3542248990 0 0) 0x7ff5fc004eb0 con 0x7ff6100c7b30 2024-03-29T11:01:52.823 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.843+0000 7ff614ff9640 1 -- 172.21.15.112:0/2105968081 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6100c7fa0 con 0x7ff6100c7b30 2024-03-29T11:01:52.823 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.843+0000 7ff614ff9640 1 -- 172.21.15.112:0/2105968081 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7ff60c0017d0 con 0x7ff6100c7b30 2024-03-29T11:01:52.823 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.843+0000 7ff614ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:52.823 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.843+0000 7ff614ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:52.823 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.843+0000 7ff614ff9640 10 monclient(hunting): dump: 2024-03-29T11:01:52.823 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:52.823 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:52.823 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:52.823 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:52.823 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:52.823 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:52.823 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:52.823 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:52.824 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:52.824 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:52.824 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.843+0000 7ff614ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:52.824 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.843+0000 7ff614ff9640 1 -- 172.21.15.112:0/2105968081 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4272893985 0 0) 0x7ff6100c5d60 con 0x7ff6100389e0 2024-03-29T11:01:52.824 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.843+0000 7ff614ff9640 1 -- 172.21.15.112:0/2105968081 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5fc004eb0 con 0x7ff6100389e0 2024-03-29T11:01:52.824 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.843+0000 7ff614ff9640 1 -- 172.21.15.112:0/2105968081 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7ff604001b70 con 0x7ff6100389e0 2024-03-29T11:01:52.824 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.843+0000 7ff614ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:52.824 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.843+0000 7ff614ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:52.824 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.843+0000 7ff614ff9640 10 monclient(hunting): dump: 2024-03-29T11:01:52.824 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:52.824 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:52.824 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:52.824 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:52.824 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:52.824 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:52.825 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:52.825 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:52.825 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:52.825 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:52.826 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.843+0000 7ff614ff9640 10 client.?.objecter ms_handle_connect 0x7ff610030660 2024-03-29T11:01:52.826 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.843+0000 7ff614ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:52.826 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff614ff9640 1 -- 172.21.15.112:0/2105968081 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2486983998 0 0) 0x7ff6100c7fa0 con 0x7ff6100c7b30 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff614ff9640 1 -- 172.21.15.112:0/2105968081 >> v1:172.21.15.118:6789/0 conn(0x7ff6100389e0 legacy=0x7ff6100c25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff614ff9640 1 -- 172.21.15.112:0/2105968081 >> v1:172.21.15.112:6789/0 conn(0x7ff610030660 legacy=0x7ff61002ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff614ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff614ff9640 1 monclient: found mon.c 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff614ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff614ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff614ff9640 1 -- 172.21.15.112:0/2105968081 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff610009fc0 con 0x7ff6100c7b30 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff614ff9640 10 monclient: _finish_auth 0 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff614ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff614ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff61d071640 5 monclient: authenticate success, global_id 4658 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff61d071640 10 monclient: _renew_subs 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff61d071640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff61d071640 1 -- 172.21.15.112:0/2105968081 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6100c7610 con 0x7ff6100c7b30 2024-03-29T11:01:52.827 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff61d071640 10 client.4658.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff61d071640 10 monclient: _renew_subs 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff61d071640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff61d071640 1 -- 172.21.15.112:0/2105968081 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff61000d360 con 0x7ff6100c7b30 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff61d071640 1 librados: init done 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff614ff9640 1 -- 172.21.15.112:0/2105968081 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff60c001cc0 con 0x7ff6100c7b30 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff614ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff614ff9640 1 -- 172.21.15.112:0/2105968081 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7ff60c0037b0 con 0x7ff6100c7b30 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff614ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff614ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff614ff9640 10 monclient: dump: 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:52.828 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:01:52.829 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:52.829 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:01:52.829 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:01:52.829 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:01:52.829 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:01:52.829 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:01:52.829 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.844+0000 7ff6167fc640 4 set_mon_vals no callback set 2024-03-29T11:01:52.829 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.846+0000 7ff6027fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.846+0000 7ff614ff9640 1 -- 172.21.15.112:0/2105968081 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7ff60c003910 con 0x7ff6100c7b30 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.847+0000 7ff614ff9640 1 -- 172.21.15.112:0/2105968081 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3769653263 0 0) 0x7ff60c0848d0 con 0x7ff6100c7b30 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.847+0000 7ff614ff9640 10 client.4658.objecter ms_dispatch 0x7ff618000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.847+0000 7ff614ff9640 3 client.4658.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.847+0000 7ff614ff9640 3 client.4658.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.847+0000 7ff614ff9640 20 client.4658.objecter dump_active .. 0 homeless 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.847+0000 7ff6027fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.847+0000 7ff6027fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.847+0000 7ff6027fc640 1 -- 172.21.15.112:0/2105968081 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff610007b90 con 0x7ff6100c7b30 2024-03-29T11:01:52.830 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.848+0000 7ff617fff640 10 monclient: get_auth_request con 0x7ff5fc06ddd0 auth_method 0 2024-03-29T11:01:52.832 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.852+0000 7ff614ff9640 10 client.4658.objecter ms_handle_connect 0x7ff5fc06ddd0 2024-03-29T11:01:52.836 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.857+0000 7ff614ff9640 1 -- 172.21.15.112:0/2105968081 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff60c0501b0 con 0x7ff6100c7b30 2024-03-29T11:01:52.837 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.857+0000 7ff614ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:52.837 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:52.857+0000 7ff614ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:53.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:53.209+0000 7ff6027fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:53.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:53.209+0000 7ff6027fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:53.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:53.209+0000 7ff6027fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-03-29T11:01:53.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:53.209+0000 7ff6027fc640 1 -- 172.21.15.112:0/2105968081 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6100c77a0 con 0x7ff6100c7b30 2024-03-29T11:01:53.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:53.210+0000 7ff614ff9640 1 -- 172.21.15.112:0/2105968081 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4236594938) 0x7ff60c0520e0 con 0x7ff6100c7b30 2024-03-29T11:01:53.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:53.210+0000 7ff614ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:53.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:53.210+0000 7ff614ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:53.190 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T11:01:53.190 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":11,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T11:01:14.980235+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:01:14.980235+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:53.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:53.213+0000 7ff6027fc640 10 librados: watch_flush enter 2024-03-29T11:01:53.194 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:53.213+0000 7ff6027fc640 10 librados: watch_flush exit 2024-03-29T11:01:53.194 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:53.213+0000 7ff6027fc640 20 client.4658.objecter shutdown clearing up homeless session... 2024-03-29T11:01:53.194 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:53.213+0000 7ff6027fc640 10 client.4658.objecter successfully canceled tick 2024-03-29T11:01:53.194 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:53.213+0000 7ff6027fc640 1 -- 172.21.15.112:0/2105968081 >> v1:172.21.15.118:6816/3181073408 conn(0x7ff5fc06ddd0 legacy=0x7ff5fc070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:53.194 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:53.213+0000 7ff6027fc640 10 monclient: shutdown 2024-03-29T11:01:53.194 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:53.213+0000 7ff6027fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:53.194 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:53.213+0000 7ff6027fc640 1 -- 172.21.15.112:0/2105968081 >> v1:172.21.15.112:6790/0 conn(0x7ff6100c7b30 legacy=0x7ff6100c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:53.194 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:53.214+0000 7ff6027fc640 1 -- 172.21.15.112:0/2105968081 shutdown_connections 2024-03-29T11:01:53.194 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:53.214+0000 7ff6027fc640 1 -- 172.21.15.112:0/2105968081 >> 172.21.15.112:0/2105968081 conn(0x7ff610005c00 msgr2=0x7ff6100c2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:53.194 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:53.214+0000 7ff6027fc640 1 -- 172.21.15.112:0/2105968081 shutdown_connections 2024-03-29T11:01:53.194 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:53.214+0000 7ff6027fc640 1 -- 172.21.15.112:0/2105968081 wait complete. 2024-03-29T11:01:53.194 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:01:53.214+0000 7ff6027fc640 1 librados: shutdown 2024-03-29T11:01:53.209 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:00.016 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:00.163 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.183+0000 7fd05f190640 10 monclient: get_monmap_and_config 2024-03-29T11:02:00.164 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.184+0000 7fd05f190640 10 monclient: build_initial_monmap 2024-03-29T11:02:00.164 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.184+0000 7fd05f190640 10 monclient: monmap: 2024-03-29T11:02:00.164 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:02:00.164 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:02:00.164 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T11:02:00.184948+0000 2024-03-29T11:02:00.164 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T11:02:00.184948+0000 2024-03-29T11:02:00.164 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:00.164 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:02:00.164 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:02:00.164 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T11:02:00.164 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T11:02:00.164 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:02:00.165 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.186+0000 7fd05f190640 1 Processor -- start 2024-03-29T11:02:00.165 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.186+0000 7fd05f190640 1 -- start start 2024-03-29T11:02:00.165 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.186+0000 7fd05f190640 10 monclient: init 2024-03-29T11:02:00.166 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.187+0000 7fd05f190640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:00.166 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.187+0000 7fd05f190640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:02:00.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.187+0000 7fd05f190640 10 monclient(hunting): picked mon.noname-b con 0x7fd0581457f0 addr v1:172.21.15.112:6790/0 2024-03-29T11:02:00.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.187+0000 7fd05f190640 10 monclient(hunting): picked mon.noname-c con 0x7fd05813f650 addr v1:172.21.15.118:6789/0 2024-03-29T11:02:00.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.187+0000 7fd05f190640 10 monclient(hunting): picked mon.noname-a con 0x7fd0580a9d70 addr v1:172.21.15.112:6789/0 2024-03-29T11:02:00.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.187+0000 7fd05f190640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0580aaef0 con 0x7fd0580a9d70 2024-03-29T11:02:00.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.187+0000 7fd05f190640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd058145bd0 con 0x7fd05813f650 2024-03-29T11:02:00.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.187+0000 7fd05f190640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd058145eb0 con 0x7fd0581457f0 2024-03-29T11:02:00.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.187+0000 7fd05f190640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:00.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.187+0000 7fd05f190640 10 monclient(hunting): authenticate will time out at 1475.491347s 2024-03-29T11:02:00.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.187+0000 7fd05cf2c640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fd0581457f0 0x7fd05813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37972/0 (socket says 172.21.15.112:37972) 2024-03-29T11:02:00.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.187+0000 7fd05cf2c640 1 -- 172.21.15.112:0/182497979 learned_addr learned my addr 172.21.15.112:0/182497979 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:02:00.169 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.188+0000 7fd0577fe640 1 -- 172.21.15.112:0/182497979 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903586833 0 0) 0x7fd058145eb0 con 0x7fd0581457f0 2024-03-29T11:02:00.170 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.188+0000 7fd0577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:02:00.170 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.188+0000 7fd0577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:00.170 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.188+0000 7fd0577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:00.170 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.188+0000 7fd0577fe640 10 monclient(hunting): my global_id is 4661 2024-03-29T11:02:00.170 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.188+0000 7fd0577fe640 1 -- 172.21.15.112:0/182497979 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd040003520 con 0x7fd0581457f0 2024-03-29T11:02:00.171 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.191+0000 7fd0577fe640 1 -- 172.21.15.112:0/182497979 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1806921896 0 0) 0x7fd040003520 con 0x7fd0581457f0 2024-03-29T11:02:00.171 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.191+0000 7fd0577fe640 1 -- 172.21.15.112:0/182497979 >> v1:172.21.15.118:6789/0 conn(0x7fd05813f650 legacy=0x7fd0580b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:02:00.171 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.191+0000 7fd0577fe640 1 -- 172.21.15.112:0/182497979 >> v1:172.21.15.112:6789/0 conn(0x7fd0580a9d70 legacy=0x7fd05814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:02:00.173 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.193+0000 7fd0577fe640 10 monclient: _finish_hunting 0 2024-03-29T11:02:00.173 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.193+0000 7fd0577fe640 1 monclient: found mon.noname-b 2024-03-29T11:02:00.173 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.193+0000 7fd0577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:00.173 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.193+0000 7fd0577fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-03-29T11:02:00.173 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.193+0000 7fd0577fe640 1 -- 172.21.15.112:0/182497979 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd058146340 con 0x7fd0581457f0 2024-03-29T11:02:00.174 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.193+0000 7fd0577fe640 10 monclient: _finish_auth 0 2024-03-29T11:02:00.174 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.193+0000 7fd0577fe640 10 monclient: _check_auth_tickets 2024-03-29T11:02:00.174 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.193+0000 7fd0577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:00.174 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.193+0000 7fd0577fe640 1 -- 172.21.15.112:0/182497979 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fd04c001690 con 0x7fd0581457f0 2024-03-29T11:02:00.174 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.193+0000 7fd0577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:00.174 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.193+0000 7fd0577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:00.174 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.193+0000 7fd0577fe640 10 monclient: dump: 2024-03-29T11:02:00.174 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:02:00.174 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:02:00.174 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:02:00.174 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:02:00.174 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:00.174 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:02:00.175 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:02:00.175 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:02:00.175 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:02:00.175 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:02:00.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.193+0000 7fd0577fe640 1 -- 172.21.15.112:0/182497979 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd04c0018f0 con 0x7fd0581457f0 2024-03-29T11:02:00.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.193+0000 7fd0577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:00.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.193+0000 7fd0577fe640 1 -- 172.21.15.112:0/182497979 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fd04c001d40 con 0x7fd0581457f0 2024-03-29T11:02:00.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.193+0000 7fd0577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:00.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.193+0000 7fd0577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:00.175 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.193+0000 7fd0577fe640 10 monclient: dump: 2024-03-29T11:02:00.175 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:02:00.175 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:02:00.176 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:02:00.176 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:02:00.176 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:00.176 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:02:00.176 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:02:00.176 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:02:00.176 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:02:00.176 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:02:00.178 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.194+0000 7fd05f190640 5 monclient: authenticate success, global_id 4661 2024-03-29T11:02:00.178 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.194+0000 7fd05f190640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:00.178 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.194+0000 7fd05f190640 4 set_mon_vals no callback set 2024-03-29T11:02:00.178 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.195+0000 7fd05f190640 10 monclient: shutdown 2024-03-29T11:02:00.178 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.195+0000 7fd05f190640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:00.178 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.195+0000 7fd05f190640 1 -- 172.21.15.112:0/182497979 >> v1:172.21.15.112:6790/0 conn(0x7fd0581457f0 legacy=0x7fd05813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:00.179 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.195+0000 7fd05f190640 1 -- 172.21.15.112:0/182497979 shutdown_connections 2024-03-29T11:02:00.179 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.195+0000 7fd05f190640 1 -- 172.21.15.112:0/182497979 >> 172.21.15.112:0/182497979 conn(0x7fd058066a00 msgr2=0x7fd058066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:00.179 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.195+0000 7fd05f190640 1 -- 172.21.15.112:0/182497979 shutdown_connections 2024-03-29T11:02:00.179 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.195+0000 7fd05f190640 1 -- 172.21.15.112:0/182497979 wait complete. 2024-03-29T11:02:00.179 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.196+0000 7fd05f190640 10 monclient: build_initial_monmap 2024-03-29T11:02:00.179 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.196+0000 7fd05f190640 10 monclient: monmap: 2024-03-29T11:02:00.179 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:02:00.179 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:00.179 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T11:02:00.179 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T11:02:00.180 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:00.180 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:02:00.180 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:02:00.180 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T11:02:00.180 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T11:02:00.180 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:02:00.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.197+0000 7fd05f190640 1 librados: starting msgr at 2024-03-29T11:02:00.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.197+0000 7fd05f190640 1 librados: starting objecter 2024-03-29T11:02:00.180 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.197+0000 7fd05f190640 1 Processor -- start 2024-03-29T11:02:00.181 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.199+0000 7fd05f190640 1 -- start start 2024-03-29T11:02:00.181 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.199+0000 7fd05f190640 1 librados: setting wanted keys 2024-03-29T11:02:00.181 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.199+0000 7fd05f190640 1 librados: calling monclient init 2024-03-29T11:02:00.181 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.199+0000 7fd05f190640 10 monclient: init 2024-03-29T11:02:00.182 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.200+0000 7fd05f190640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:00.182 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.200+0000 7fd05f190640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:02:00.182 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.200+0000 7fd05f190640 10 monclient(hunting): picked mon.noname-b con 0x7fd0580a9d70 addr v1:172.21.15.118:6789/0 2024-03-29T11:02:00.182 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.200+0000 7fd05f190640 10 monclient(hunting): picked mon.noname-c con 0x7fd05813f650 addr v1:172.21.15.112:6790/0 2024-03-29T11:02:00.182 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.200+0000 7fd05f190640 10 monclient(hunting): picked mon.noname-a con 0x7fd0581457f0 addr v1:172.21.15.112:6789/0 2024-03-29T11:02:00.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.200+0000 7fd05f190640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd05813df00 con 0x7fd0581457f0 2024-03-29T11:02:00.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.200+0000 7fd05f190640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd058145200 con 0x7fd0580a9d70 2024-03-29T11:02:00.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.200+0000 7fd05f190640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd05813da70 con 0x7fd05813f650 2024-03-29T11:02:00.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.200+0000 7fd05f190640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:00.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.200+0000 7fd05f190640 10 monclient(hunting): authenticate will time out at 1475.504352s 2024-03-29T11:02:00.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.200+0000 7fd057fff640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fd05813f650 0x7fd058142210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:38670/0 (socket says 172.21.15.112:38670) 2024-03-29T11:02:00.183 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.200+0000 7fd057fff640 1 -- 172.21.15.112:0/1526918354 learned_addr learned my addr 172.21.15.112:0/1526918354 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:02:00.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 10 client.?.objecter ms_handle_connect 0x7fd05813f650 2024-03-29T11:02:00.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:00.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 10 client.?.objecter ms_handle_connect 0x7fd0580a9d70 2024-03-29T11:02:00.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:00.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 1 -- 172.21.15.112:0/1526918354 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1886856253 0 0) 0x7fd058145200 con 0x7fd0580a9d70 2024-03-29T11:02:00.186 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 10 monclient(hunting): _negotiate 2024-03-29T11:02:00.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:00.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:00.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 10 monclient(hunting): my global_id is 4573 2024-03-29T11:02:00.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 1 -- 172.21.15.112:0/1526918354 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd044004eb0 con 0x7fd0580a9d70 2024-03-29T11:02:00.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 1 -- 172.21.15.112:0/1526918354 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3920836391 0 0) 0x7fd05813da70 con 0x7fd05813f650 2024-03-29T11:02:00.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 10 monclient(hunting): _negotiate 2024-03-29T11:02:00.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:00.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:00.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 10 monclient(hunting): my global_id is 4664 2024-03-29T11:02:00.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 1 -- 172.21.15.112:0/1526918354 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd058145200 con 0x7fd05813f650 2024-03-29T11:02:00.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 10 client.?.objecter ms_handle_connect 0x7fd0581457f0 2024-03-29T11:02:00.187 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:00.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 1 -- 172.21.15.112:0/1526918354 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1553758784 0 0) 0x7fd044004eb0 con 0x7fd0580a9d70 2024-03-29T11:02:00.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 1 -- 172.21.15.112:0/1526918354 --> v1:172.21.15.118:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd05813da70 con 0x7fd0580a9d70 2024-03-29T11:02:00.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 1 -- 172.21.15.112:0/1526918354 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fd04c001d40 con 0x7fd0580a9d70 2024-03-29T11:02:00.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:00.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:00.188 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 10 monclient(hunting): dump: 2024-03-29T11:02:00.188 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:02:00.188 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:02:00.188 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:02:00.188 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:02:00.188 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:00.188 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:02:00.188 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:02:00.189 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:02:00.189 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:02:00.189 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:02:00.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:00.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 1 -- 172.21.15.112:0/1526918354 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422361504 0 0) 0x7fd05813df00 con 0x7fd0581457f0 2024-03-29T11:02:00.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 10 monclient(hunting): _negotiate 2024-03-29T11:02:00.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:00.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:00.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 10 monclient(hunting): my global_id is 4578 2024-03-29T11:02:00.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 1 -- 172.21.15.112:0/1526918354 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd044004eb0 con 0x7fd0581457f0 2024-03-29T11:02:00.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.203+0000 7fd0557fa640 1 -- 172.21.15.112:0/1526918354 <== mon.1 v1:172.21.15.118:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3241517614 0 0) 0x7fd05813da70 con 0x7fd0580a9d70 2024-03-29T11:02:00.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd0557fa640 1 -- 172.21.15.112:0/1526918354 >> v1:172.21.15.112:6790/0 conn(0x7fd05813f650 legacy=0x7fd058142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:00.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd0557fa640 1 -- 172.21.15.112:0/1526918354 >> v1:172.21.15.112:6789/0 conn(0x7fd0581457f0 legacy=0x7fd0581de340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:00.189 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd0557fa640 10 monclient: _finish_hunting 0 2024-03-29T11:02:00.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd0557fa640 1 monclient: found mon.b 2024-03-29T11:02:00.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd0557fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:00.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd0557fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:02:00.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd0557fa640 1 -- 172.21.15.112:0/1526918354 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05813dc00 con 0x7fd0580a9d70 2024-03-29T11:02:00.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd0557fa640 10 monclient: _finish_auth 0 2024-03-29T11:02:00.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd0557fa640 10 monclient: _check_auth_tickets 2024-03-29T11:02:00.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd0557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:00.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd0557fa640 1 -- 172.21.15.112:0/1526918354 <== mon.1 v1:172.21.15.118:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd04c004b40 con 0x7fd0580a9d70 2024-03-29T11:02:00.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd0557fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:00.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd0557fa640 1 -- 172.21.15.112:0/1526918354 <== mon.1 v1:172.21.15.118:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7fd04c004e50 con 0x7fd0580a9d70 2024-03-29T11:02:00.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd0557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:00.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd0557fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:02:00.190 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd0557fa640 10 monclient: dump: 2024-03-29T11:02:00.191 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:02:00.191 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:02:00.191 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:02:00.191 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:02:00.191 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:00.191 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:02:00.191 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:02:00.191 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:02:00.191 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:02:00.191 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:02:00.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd05f190640 5 monclient: authenticate success, global_id 4573 2024-03-29T11:02:00.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd05f190640 10 monclient: _renew_subs 2024-03-29T11:02:00.191 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd05f190640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:02:00.192 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd05f190640 1 -- 172.21.15.112:0/1526918354 --> v1:172.21.15.118:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd058140680 con 0x7fd0580a9d70 2024-03-29T11:02:00.192 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd0567fc640 4 set_mon_vals no callback set 2024-03-29T11:02:00.192 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd05f190640 10 client.4573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:00.192 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd05f190640 10 monclient: _renew_subs 2024-03-29T11:02:00.192 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd05f190640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:02:00.192 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd05f190640 1 -- 172.21.15.112:0/1526918354 --> v1:172.21.15.118:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0581d3890 con 0x7fd0580a9d70 2024-03-29T11:02:00.192 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.204+0000 7fd05f190640 1 librados: init done 2024-03-29T11:02:00.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.207+0000 7fd0557fa640 1 -- 172.21.15.112:0/1526918354 <== mon.1 v1:172.21.15.118:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7fd04c01ae20 con 0x7fd0580a9d70 2024-03-29T11:02:00.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.208+0000 7fd0557fa640 1 -- 172.21.15.112:0/1526918354 <== mon.1 v1:172.21.15.118:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3769653263 0 0) 0x7fd04c084ad0 con 0x7fd0580a9d70 2024-03-29T11:02:00.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.208+0000 7fd0557fa640 10 client.4573.objecter ms_dispatch 0x7fd058000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:02:00.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.208+0000 7fd0557fa640 3 client.4573.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:02:00.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.208+0000 7fd0557fa640 3 client.4573.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:02:00.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.208+0000 7fd0557fa640 20 client.4573.objecter dump_active .. 0 homeless 2024-03-29T11:02:00.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.208+0000 7fd057fff640 10 monclient: get_auth_request con 0x7fd04406de50 auth_method 0 2024-03-29T11:02:00.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.208+0000 7fd05f190640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:00.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.208+0000 7fd05f190640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:00.193 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.208+0000 7fd05f190640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:02:00.194 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.208+0000 7fd05f190640 1 -- 172.21.15.112:0/1526918354 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd05813e090 con 0x7fd0580a9d70 2024-03-29T11:02:00.194 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.208+0000 7fd0557fa640 10 client.4573.objecter ms_handle_connect 0x7fd04406de50 2024-03-29T11:02:00.196 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.217+0000 7fd0557fa640 1 -- 172.21.15.112:0/1526918354 <== mon.1 v1:172.21.15.118:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd04c0503b0 con 0x7fd0580a9d70 2024-03-29T11:02:00.196 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.217+0000 7fd0557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:00.196 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.217+0000 7fd0557fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:00.529 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.549+0000 7fd05f190640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:00.529 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.549+0000 7fd05f190640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:00.529 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.549+0000 7fd05f190640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.118:6789/0 2024-03-29T11:02:00.529 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.549+0000 7fd05f190640 1 -- 172.21.15.112:0/1526918354 --> v1:172.21.15.118:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd058141d10 con 0x7fd0580a9d70 2024-03-29T11:02:00.530 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.550+0000 7fd0557fa640 1 -- 172.21.15.112:0/1526918354 <== mon.1 v1:172.21.15.118:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4236594938) 0x7fd04c0522e0 con 0x7fd0580a9d70 2024-03-29T11:02:00.530 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.550+0000 7fd0557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:00.530 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.550+0000 7fd0557fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:00.530 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T11:02:00.530 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":11,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T11:01:14.980235+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:01:14.980235+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:00.534 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.554+0000 7fd05f190640 10 librados: watch_flush enter 2024-03-29T11:02:00.534 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.554+0000 7fd05f190640 10 librados: watch_flush exit 2024-03-29T11:02:00.534 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.554+0000 7fd05f190640 20 client.4573.objecter shutdown clearing up homeless session... 2024-03-29T11:02:00.534 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.554+0000 7fd05f190640 10 client.4573.objecter successfully canceled tick 2024-03-29T11:02:00.534 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.554+0000 7fd05f190640 1 -- 172.21.15.112:0/1526918354 >> v1:172.21.15.118:6816/3181073408 conn(0x7fd04406de50 legacy=0x7fd044070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:00.534 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.554+0000 7fd05f190640 10 monclient: shutdown 2024-03-29T11:02:00.534 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.554+0000 7fd05f190640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:00.535 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.554+0000 7fd05f190640 1 -- 172.21.15.112:0/1526918354 >> v1:172.21.15.118:6789/0 conn(0x7fd0580a9d70 legacy=0x7fd0581d2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:00.535 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.554+0000 7fd05f190640 1 -- 172.21.15.112:0/1526918354 shutdown_connections 2024-03-29T11:02:00.535 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.554+0000 7fd05f190640 1 -- 172.21.15.112:0/1526918354 >> 172.21.15.112:0/1526918354 conn(0x7fd058066a00 msgr2=0x7fd058137260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:00.535 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.555+0000 7fd05f190640 1 -- 172.21.15.112:0/1526918354 shutdown_connections 2024-03-29T11:02:00.535 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.555+0000 7fd05f190640 1 -- 172.21.15.112:0/1526918354 wait complete. 2024-03-29T11:02:00.537 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:00.557+0000 7fd05f190640 1 librados: shutdown 2024-03-29T11:02:00.554 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:07.360 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:07.497 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.516+0000 7f6415486640 10 monclient: get_monmap_and_config 2024-03-29T11:02:07.498 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.516+0000 7f6415486640 10 monclient: build_initial_monmap 2024-03-29T11:02:07.498 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.516+0000 7f6415486640 10 monclient: monmap: 2024-03-29T11:02:07.498 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:02:07.498 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:02:07.498 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T11:02:07.518627+0000 2024-03-29T11:02:07.498 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T11:02:07.518627+0000 2024-03-29T11:02:07.498 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:07.498 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:02:07.498 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:02:07.498 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c 2024-03-29T11:02:07.498 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-03-29T11:02:07.498 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:02:07.500 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.518+0000 7f6415486640 1 Processor -- start 2024-03-29T11:02:07.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.518+0000 7f6415486640 1 -- start start 2024-03-29T11:02:07.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.518+0000 7f6415486640 10 monclient: init 2024-03-29T11:02:07.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.518+0000 7f6415486640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:07.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.519+0000 7f6415486640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:02:07.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.519+0000 7f6415486640 10 monclient(hunting): picked mon.noname-b con 0x7f64100b4100 addr v1:172.21.15.112:6790/0 2024-03-29T11:02:07.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.519+0000 7f6415486640 10 monclient(hunting): picked mon.noname-a con 0x7f641013ee90 addr v1:172.21.15.112:6789/0 2024-03-29T11:02:07.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.519+0000 7f6415486640 10 monclient(hunting): picked mon.noname-c con 0x7f64100a9ac0 addr v1:172.21.15.118:6789/0 2024-03-29T11:02:07.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.519+0000 7f6415486640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64100af6f0 con 0x7f641013ee90 2024-03-29T11:02:07.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.519+0000 7f6415486640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64100aac80 con 0x7f64100a9ac0 2024-03-29T11:02:07.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.519+0000 7f6415486640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64100aaf60 con 0x7f64100b4100 2024-03-29T11:02:07.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.519+0000 7f6415486640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:07.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.519+0000 7f6415486640 10 monclient(hunting): authenticate will time out at 1482.824281s 2024-03-29T11:02:07.501 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.519+0000 7f640e7fc640 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f641013ee90 0x7f641013b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:39572/0 (socket says 172.21.15.112:39572) 2024-03-29T11:02:07.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.519+0000 7f640e7fc640 1 -- 172.21.15.112:0/724979807 learned_addr learned my addr 172.21.15.112:0/724979807 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:02:07.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.519+0000 7f640dffb640 1 -- 172.21.15.112:0/724979807 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153629765 0 0) 0x7f64100af6f0 con 0x7f641013ee90 2024-03-29T11:02:07.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.519+0000 7f640dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:02:07.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.519+0000 7f640dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:07.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.519+0000 7f640dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:07.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.519+0000 7f640dffb640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:02:07.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.519+0000 7f640dffb640 1 -- 172.21.15.112:0/724979807 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63f8003520 con 0x7f641013ee90 2024-03-29T11:02:07.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.519+0000 7f640dffb640 1 -- 172.21.15.112:0/724979807 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 136144355 0 0) 0x7f63f8003520 con 0x7f641013ee90 2024-03-29T11:02:07.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.519+0000 7f640dffb640 1 -- 172.21.15.112:0/724979807 >> v1:172.21.15.112:6790/0 conn(0x7f64100b4100 legacy=0x7f64100adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:02:07.502 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.519+0000 7f640dffb640 1 -- 172.21.15.112:0/724979807 >> v1:172.21.15.118:6789/0 conn(0x7f64100a9ac0 legacy=0x7f641014ac10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:02:07.503 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.522+0000 7f640dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:02:07.503 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.522+0000 7f640dffb640 1 monclient: found mon.noname-a 2024-03-29T11:02:07.503 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.522+0000 7f640dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:07.503 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.522+0000 7f640dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-03-29T11:02:07.503 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.522+0000 7f640dffb640 1 -- 172.21.15.112:0/724979807 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641013e970 con 0x7f641013ee90 2024-03-29T11:02:07.503 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.522+0000 7f640dffb640 10 monclient: _finish_auth 0 2024-03-29T11:02:07.503 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.522+0000 7f640dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:02:07.503 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.522+0000 7f640dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:07.504 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.522+0000 7f640dffb640 1 -- 172.21.15.112:0/724979807 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f6400001670 con 0x7f641013ee90 2024-03-29T11:02:07.504 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.522+0000 7f640dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:07.504 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.522+0000 7f640dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:02:07.504 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.522+0000 7f640dffb640 10 monclient: dump: 2024-03-29T11:02:07.504 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:02:07.504 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:02:07.504 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:02:07.504 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:02:07.504 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:07.504 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:02:07.504 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:02:07.504 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:02:07.504 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:02:07.504 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:02:07.505 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.522+0000 7f6415486640 5 monclient: authenticate success, global_id 4581 2024-03-29T11:02:07.505 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.522+0000 7f6415486640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:07.506 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.523+0000 7f640dffb640 1 -- 172.21.15.112:0/724979807 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64000018d0 con 0x7f641013ee90 2024-03-29T11:02:07.506 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.523+0000 7f640dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:07.506 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.523+0000 7f6415486640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:07.506 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.523+0000 7f6415486640 4 set_mon_vals no callback set 2024-03-29T11:02:07.506 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.524+0000 7f640dffb640 1 -- 172.21.15.112:0/724979807 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f6400001d20 con 0x7f641013ee90 2024-03-29T11:02:07.506 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.524+0000 7f640dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:07.506 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.524+0000 7f640dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:07.506 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.524+0000 7f640dffb640 10 monclient: dump: 2024-03-29T11:02:07.506 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:02:07.506 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:02:07.507 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:02:07.507 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:02:07.507 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:07.507 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:02:07.507 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:02:07.507 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:02:07.507 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:02:07.507 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:02:07.507 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.524+0000 7f6415486640 10 monclient: shutdown 2024-03-29T11:02:07.507 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.524+0000 7f6415486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:07.507 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.524+0000 7f6415486640 1 -- 172.21.15.112:0/724979807 >> v1:172.21.15.112:6789/0 conn(0x7f641013ee90 legacy=0x7f641013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:07.507 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.524+0000 7f6415486640 1 -- 172.21.15.112:0/724979807 shutdown_connections 2024-03-29T11:02:07.507 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.524+0000 7f6415486640 1 -- 172.21.15.112:0/724979807 >> 172.21.15.112:0/724979807 conn(0x7f64100b35e0 msgr2=0x7f64100b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:07.507 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.524+0000 7f6415486640 1 -- 172.21.15.112:0/724979807 shutdown_connections 2024-03-29T11:02:07.508 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.524+0000 7f6415486640 1 -- 172.21.15.112:0/724979807 wait complete. 2024-03-29T11:02:07.508 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.525+0000 7f6415486640 10 monclient: build_initial_monmap 2024-03-29T11:02:07.508 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.525+0000 7f6415486640 10 monclient: monmap: 2024-03-29T11:02:07.508 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-03-29T11:02:07.508 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:07.508 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-03-29T11:02:07.508 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-03-29T11:02:07.508 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:07.508 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:02:07.508 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-03-29T11:02:07.508 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.noname-b 2024-03-29T11:02:07.508 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-03-29T11:02:07.508 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:02:07.513 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.526+0000 7f6415486640 1 librados: starting msgr at 2024-03-29T11:02:07.513 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.526+0000 7f6415486640 1 librados: starting objecter 2024-03-29T11:02:07.513 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.526+0000 7f6415486640 1 Processor -- start 2024-03-29T11:02:07.513 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.526+0000 7f6415486640 1 -- start start 2024-03-29T11:02:07.513 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.526+0000 7f6415486640 1 librados: setting wanted keys 2024-03-29T11:02:07.513 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.526+0000 7f6415486640 1 librados: calling monclient init 2024-03-29T11:02:07.514 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.526+0000 7f6415486640 10 monclient: init 2024-03-29T11:02:07.514 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.527+0000 7f6415486640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:07.514 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.527+0000 7f6415486640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:02:07.514 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.527+0000 7f6415486640 10 monclient(hunting): picked mon.noname-b con 0x7f64100a9ac0 addr v1:172.21.15.118:6789/0 2024-03-29T11:02:07.514 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.527+0000 7f6415486640 10 monclient(hunting): picked mon.noname-a con 0x7f64100b4100 addr v1:172.21.15.112:6789/0 2024-03-29T11:02:07.514 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.528+0000 7f6415486640 10 monclient(hunting): picked mon.noname-c con 0x7f641013ee90 addr v1:172.21.15.112:6790/0 2024-03-29T11:02:07.514 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.528+0000 7f6415486640 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6410071fb0 con 0x7f64100b4100 2024-03-29T11:02:07.514 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.528+0000 7f6415486640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6410072200 con 0x7f64100a9ac0 2024-03-29T11:02:07.514 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.528+0000 7f6415486640 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6410070ba0 con 0x7f641013ee90 2024-03-29T11:02:07.514 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.528+0000 7f6415486640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:07.514 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.528+0000 7f6415486640 10 monclient(hunting): authenticate will time out at 1482.833368s 2024-03-29T11:02:07.514 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.528+0000 7f640f7fe640 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f641013ee90 0x7f64101e70c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:60808/0 (socket says 172.21.15.112:60808) 2024-03-29T11:02:07.515 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.528+0000 7f640f7fe640 1 -- 172.21.15.112:0/1927415300 learned_addr learned my addr 172.21.15.112:0/1927415300 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-03-29T11:02:07.515 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.528+0000 7f63f77fe640 10 client.?.objecter ms_handle_connect 0x7f641013ee90 2024-03-29T11:02:07.515 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.528+0000 7f63f77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:07.515 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.529+0000 7f63f77fe640 10 client.?.objecter ms_handle_connect 0x7f64100b4100 2024-03-29T11:02:07.515 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.529+0000 7f63f77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:07.515 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.529+0000 7f63f77fe640 1 -- 172.21.15.112:0/1927415300 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659090119 0 0) 0x7f6410070ba0 con 0x7f641013ee90 2024-03-29T11:02:07.515 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.529+0000 7f63f77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:02:07.515 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.529+0000 7f63f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:07.515 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.529+0000 7f63f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:07.515 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.529+0000 7f63f77fe640 10 monclient(hunting): my global_id is 4667 2024-03-29T11:02:07.516 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.529+0000 7f63f77fe640 1 -- 172.21.15.112:0/1927415300 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63fc004eb0 con 0x7f641013ee90 2024-03-29T11:02:07.516 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.529+0000 7f63f77fe640 1 -- 172.21.15.112:0/1927415300 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011033025 0 0) 0x7f6410071fb0 con 0x7f64100b4100 2024-03-29T11:02:07.516 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.529+0000 7f63f77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:02:07.516 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.529+0000 7f63f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:07.516 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.529+0000 7f63f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:07.517 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.529+0000 7f63f77fe640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:02:07.517 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.529+0000 7f63f77fe640 1 -- 172.21.15.112:0/1927415300 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6410070ba0 con 0x7f64100b4100 2024-03-29T11:02:07.517 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.529+0000 7f63f77fe640 1 -- 172.21.15.112:0/1927415300 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2093665215 0 0) 0x7f6410070ba0 con 0x7f64100b4100 2024-03-29T11:02:07.517 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.529+0000 7f63f77fe640 1 -- 172.21.15.112:0/1927415300 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6410071fb0 con 0x7f64100b4100 2024-03-29T11:02:07.517 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.529+0000 7f63f77fe640 1 -- 172.21.15.112:0/1927415300 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f6400001630 con 0x7f64100b4100 2024-03-29T11:02:07.517 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.529+0000 7f63f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:07.517 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.529+0000 7f63f77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:02:07.517 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.529+0000 7f63f77fe640 10 monclient(hunting): dump: 2024-03-29T11:02:07.517 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:02:07.517 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:02:07.517 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:02:07.517 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:02:07.518 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:07.518 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:02:07.518 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:02:07.518 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:02:07.518 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:02:07.518 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:02:07.518 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.529+0000 7f63f77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:07.518 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.529+0000 7f63f77fe640 1 -- 172.21.15.112:0/1927415300 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 395479919 0 0) 0x7f6410071fb0 con 0x7f64100b4100 2024-03-29T11:02:07.518 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.529+0000 7f63f77fe640 1 -- 172.21.15.112:0/1927415300 >> v1:172.21.15.112:6790/0 conn(0x7f641013ee90 legacy=0x7f64101e70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:07.518 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.529+0000 7f63f77fe640 1 -- 172.21.15.112:0/1927415300 >> v1:172.21.15.118:6789/0 conn(0x7f64100a9ac0 legacy=0x7f641013b000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:02:07.518 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.531+0000 7f63f77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:02:07.518 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.531+0000 7f63f77fe640 1 monclient: found mon.a 2024-03-29T11:02:07.519 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.531+0000 7f63f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:07.519 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.531+0000 7f63f77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T11:02:07.519 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.531+0000 7f63f77fe640 1 -- 172.21.15.112:0/1927415300 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64100710c0 con 0x7f64100b4100 2024-03-29T11:02:07.519 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.531+0000 7f63f77fe640 10 monclient: _finish_auth 0 2024-03-29T11:02:07.519 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.531+0000 7f63f77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:02:07.519 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.531+0000 7f63f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:07.519 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.531+0000 7f63f77fe640 1 -- 172.21.15.112:0/1927415300 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6400004b00 con 0x7f64100b4100 2024-03-29T11:02:07.519 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.531+0000 7f63f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:07.519 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.531+0000 7f640d7fa640 4 set_mon_vals no callback set 2024-03-29T11:02:07.519 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.531+0000 7f63f77fe640 1 -- 172.21.15.112:0/1927415300 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2757563083 0 0) 0x7f6400004ed0 con 0x7f64100b4100 2024-03-29T11:02:07.519 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.531+0000 7f63f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:07.519 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.531+0000 7f63f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:07.520 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.531+0000 7f63f77fe640 10 monclient: dump: 2024-03-29T11:02:07.520 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-03-29T11:02:07.520 INFO:teuthology.misc.health.smithi112.stderr:fsid a3151ab9-9e71-4afa-ac98-17b78e4993d2 2024-03-29T11:02:07.520 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-03-29T10:50:33.381725+0000 2024-03-29T11:02:07.520 INFO:teuthology.misc.health.smithi112.stderr:created 2024-03-29T10:50:33.381725+0000 2024-03-29T11:02:07.520 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:07.520 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-03-29T11:02:07.520 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-03-29T11:02:07.520 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.118:6789/0 mon.b 2024-03-29T11:02:07.520 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-03-29T11:02:07.520 INFO:teuthology.misc.health.smithi112.stderr: 2024-03-29T11:02:07.520 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.531+0000 7f6415486640 5 monclient: authenticate success, global_id 4584 2024-03-29T11:02:07.520 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.531+0000 7f6415486640 10 monclient: _renew_subs 2024-03-29T11:02:07.521 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.531+0000 7f6415486640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T11:02:07.521 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.531+0000 7f6415486640 1 -- 172.21.15.112:0/1927415300 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f641006fb80 con 0x7f64100b4100 2024-03-29T11:02:07.521 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.531+0000 7f6415486640 10 client.4584.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:07.521 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.531+0000 7f6415486640 10 monclient: _renew_subs 2024-03-29T11:02:07.521 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.531+0000 7f6415486640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T11:02:07.521 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.531+0000 7f6415486640 1 -- 172.21.15.112:0/1927415300 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64100708a0 con 0x7f64100b4100 2024-03-29T11:02:07.521 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.531+0000 7f6415486640 1 librados: init done 2024-03-29T11:02:07.522 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.534+0000 7f63f77fe640 1 -- 172.21.15.112:0/1927415300 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1551855696 0 0) 0x7f6400005030 con 0x7f64100b4100 2024-03-29T11:02:07.523 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.534+0000 7f6415486640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:07.523 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.534+0000 7f6415486640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:07.523 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.534+0000 7f6415486640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T11:02:07.523 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.534+0000 7f6415486640 1 -- 172.21.15.112:0/1927415300 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64100af900 con 0x7f64100b4100 2024-03-29T11:02:07.523 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.535+0000 7f63f77fe640 1 -- 172.21.15.112:0/1927415300 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3769653263 0 0) 0x7f6400084c00 con 0x7f64100b4100 2024-03-29T11:02:07.523 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.535+0000 7f63f77fe640 10 client.4584.objecter ms_dispatch 0x7f6410000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:02:07.523 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.535+0000 7f63f77fe640 3 client.4584.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:02:07.523 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.535+0000 7f63f77fe640 3 client.4584.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:02:07.523 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.535+0000 7f63f77fe640 20 client.4584.objecter dump_active .. 0 homeless 2024-03-29T11:02:07.523 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.535+0000 7f640effd640 10 monclient: get_auth_request con 0x7f63fc06dd30 auth_method 0 2024-03-29T11:02:07.523 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.535+0000 7f63f77fe640 10 client.4584.objecter ms_handle_connect 0x7f63fc06dd30 2024-03-29T11:02:07.524 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.542+0000 7f63f77fe640 1 -- 172.21.15.112:0/1927415300 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f64100af900 con 0x7f64100b4100 2024-03-29T11:02:07.524 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.542+0000 7f63f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:07.524 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.542+0000 7f63f77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:07.844 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.864+0000 7f6415486640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:07.844 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.864+0000 7f6415486640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:07.844 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.864+0000 7f6415486640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-03-29T11:02:07.844 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.864+0000 7f6415486640 1 -- 172.21.15.112:0/1927415300 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64100af900 con 0x7f64100b4100 2024-03-29T11:02:07.847 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.868+0000 7f63f77fe640 1 -- 172.21.15.112:0/1927415300 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4236594938) 0x7f64100af900 con 0x7f64100b4100 2024-03-29T11:02:07.847 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.868+0000 7f63f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:07.847 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.868+0000 7f63f77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:07.850 INFO:teuthology.misc.health.smithi112.stdout: 2024-03-29T11:02:07.850 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":11,"fsid":"a3151ab9-9e71-4afa-ac98-17b78e4993d2","created":"2024-03-29T10:51:10.039684+0000","modified":"2024-03-29T11:01:14.980235+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:01:14.980235+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84afddc6-02ba-4dda-8b9a-ea9b91510482","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"600a2c50-5341-4503-847b-fd2a2922a007","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3ec4da85-e6cc-4d66-b76e-986e94597743","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"bcf148cb-34a5-47b1-9c89-9956c132153b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60b98350-a43a-40a3-8d6a-fa227192c87a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bd7c5f98-7ca2-48f7-a7d5-34d4c6d24d2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ffd83441-b00f-42ba-ba42-69f1fadb69eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f5b800e-e099-41e8-8fdc-4a5ad8828199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:07.851 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.871+0000 7f6415486640 10 librados: watch_flush enter 2024-03-29T11:02:07.851 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.871+0000 7f6415486640 10 librados: watch_flush exit 2024-03-29T11:02:07.851 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.871+0000 7f6415486640 20 client.4584.objecter shutdown clearing up homeless session... 2024-03-29T11:02:07.851 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.871+0000 7f6415486640 10 client.4584.objecter successfully canceled tick 2024-03-29T11:02:07.851 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.871+0000 7f6415486640 1 -- 172.21.15.112:0/1927415300 >> v1:172.21.15.118:6816/3181073408 conn(0x7f63fc06dd30 legacy=0x7f63fc0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:07.851 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.871+0000 7f6415486640 10 monclient: shutdown 2024-03-29T11:02:07.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.871+0000 7f6415486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:07.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.871+0000 7f6415486640 1 -- 172.21.15.112:0/1927415300 >> v1:172.21.15.112:6789/0 conn(0x7f64100b4100 legacy=0x7f64101db880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:07.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.871+0000 7f6415486640 1 -- 172.21.15.112:0/1927415300 shutdown_connections 2024-03-29T11:02:07.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.871+0000 7f6415486640 1 -- 172.21.15.112:0/1927415300 >> 172.21.15.112:0/1927415300 conn(0x7f64100b35e0 msgr2=0x7f64100b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:07.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.871+0000 7f6415486640 1 -- 172.21.15.112:0/1927415300 shutdown_connections 2024-03-29T11:02:07.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.871+0000 7f6415486640 1 -- 172.21.15.112:0/1927415300 wait complete. 2024-03-29T11:02:07.852 INFO:teuthology.misc.health.smithi112.stderr:2024-03-29T11:02:07.871+0000 7f6415486640 1 librados: shutdown 2024-03-29T11:02:07.867 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:07.868 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:02:07.869 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:02:07.870 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:02:07.870 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:07.908 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:02:07.908 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:02:07.908 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:07.988 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:02:07.988 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:02:07.988 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:08.058 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:02:08.059 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:02:08.059 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:08.126 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:02:08.126 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:02:08.126 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:08.164 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:02:08.164 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:02:08.164 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:08.239 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:02:08.240 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:02:08.240 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:08.318 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:02:08.318 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:02:08.318 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:08.390 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:02:08.390 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:02:08.391 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:02:08.391 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:08.414 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:02:08.414 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:02:08.415 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:08.456 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:02:08.456 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:02:08.456 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:02:08.456 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:08.468 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:02:08.468 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:02:08.468 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:08.520 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:02:08.520 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:02:08.520 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:08.531 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:02:08.531 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:02:08.531 DEBUG:teuthology.orchestra.run.smithi112:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:02:08.707 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi112.front.sepia.ceph.com 2024-03-29T11:02:08.707 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:02:09.636 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi112.front.sepia.ceph.com 2024-03-29T11:02:09.636 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:02:09.747 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi112.front.sepia.ceph.com 2024-03-29T11:02:09.747 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:02:09.803 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi112.front.sepia.ceph.com 2024-03-29T11:02:09.803 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:02:09.861 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi118.front.sepia.ceph.com 2024-03-29T11:02:09.861 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:02:12.726 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi118.front.sepia.ceph.com 2024-03-29T11:02:12.726 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:02:12.830 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi118.front.sepia.ceph.com 2024-03-29T11:02:12.830 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:02:12.877 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi118.front.sepia.ceph.com 2024-03-29T11:02:12.878 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:02:12.927 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:02:12.928 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622657/data/mon.a.tgz 2024-03-29T11:02:12.929 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2024-03-29T11:02:12.948 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.sQYHEmXU2Z 2024-03-29T11:02:12.949 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.sQYHEmXU2Z 2024-03-29T11:02:13.063 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.sQYHEmXU2Z 2024-03-29T11:02:13.152 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.sQYHEmXU2Z is 211KB 2024-03-29T11:02:13.248 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.sQYHEmXU2Z 2024-03-29T11:02:13.264 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622657/data/mon.c.tgz 2024-03-29T11:02:13.264 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2024-03-29T11:02:13.320 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.QzB3mkLBVC 2024-03-29T11:02:13.321 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.QzB3mkLBVC 2024-03-29T11:02:13.422 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.QzB3mkLBVC 2024-03-29T11:02:13.516 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.QzB3mkLBVC is 212KB 2024-03-29T11:02:13.584 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.QzB3mkLBVC 2024-03-29T11:02:13.600 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622657/data/mon.b.tgz 2024-03-29T11:02:13.600 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2024-03-29T11:02:13.618 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.yLXVTGsH3p 2024-03-29T11:02:13.619 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.yLXVTGsH3p 2024-03-29T11:02:13.760 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.yLXVTGsH3p 2024-03-29T11:02:13.857 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.yLXVTGsH3p is 212KB 2024-03-29T11:02:13.931 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.yLXVTGsH3p 2024-03-29T11:02:13.946 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:02:13.947 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:02:13.949 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:02:14.029 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:02:14.067 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:02:14.067 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:02:14.069 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622657/remote/smithi112/crash 2024-03-29T11:02:14.070 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:02:14.097 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622657/remote/smithi118/crash 2024-03-29T11:02:14.098 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:02:14.133 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:02:14.134 DEBUG:teuthology.orchestra.run.smithi112:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:02:14.141 DEBUG:teuthology.orchestra.run.smithi118:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:02:14.165 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38853.log 2024-03-29T11:02:14.165 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39947.log 2024-03-29T11:02:14.219 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:02:14.219 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34652.log 2024-03-29T11:02:14.219 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34612.log 2024-03-29T11:02:14.220 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34492.log 2024-03-29T11:02:14.220 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.34612.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:02:14.220 INFO:teuthology.orchestra.run.smithi118.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34652.log.gz 2024-03-29T11:02:14.220 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34572.log 2024-03-29T11:02:14.220 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.34492.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34612.log.gz 2024-03-29T11:02:14.220 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34412.log 2024-03-29T11:02:14.220 INFO:teuthology.orchestra.run.smithi118.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34492.log.gz 2024-03-29T11:02:14.220 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.34572.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:02:14.220 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34452.log 2024-03-29T11:02:14.220 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:02:14.220 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:02:14.220 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.34412.log: 88.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:02:14.220 INFO:teuthology.orchestra.run.smithi118.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34572.log.gz 2024-03-29T11:02:14.221 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.34452.log: /var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:02:14.221 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-osd.5.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34412.log.gz 2024-03-29T11:02:14.221 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34452.log.gz 2024-03-29T11:02:14.221 INFO:teuthology.orchestra.run.smithi118.stderr: -- /var/log/ceph/ceph.log 2024-03-29T11:02:14.221 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34532.log 2024-03-29T11:02:14.221 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:02:14.221 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.34532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34692.log 2024-03-29T11:02:14.221 INFO:teuthology.orchestra.run.smithi118.stderr: 87.1% 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34532.log.gz 2024-03-29T11:02:14.221 INFO:teuthology.orchestra.run.smithi118.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:02:14.221 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.34692.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34692.log.gz 2024-03-29T11:02:14.224 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28687.log 2024-03-29T11:02:14.224 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38853.log: /var/log/ceph/ceph-client.admin.39947.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:02:14.224 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph.tmp-client.admin.28687.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38853.log.gz 2024-03-29T11:02:14.224 INFO:teuthology.orchestra.run.smithi112.stderr:gzip 0.0% -5 -- replaced with /var/log/ceph/ceph.tmp-client.admin.28687.log.gz 2024-03-29T11:02:14.224 INFO:teuthology.orchestra.run.smithi112.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39243.log 2024-03-29T11:02:14.224 INFO:teuthology.orchestra.run.smithi112.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39947.log.gz 2024-03-29T11:02:14.224 INFO:teuthology.orchestra.run.smithi112.stderr:gzip/var/log/ceph/ceph-mon.a.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38463.log 2024-03-29T11:02:14.225 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38580.log 2024-03-29T11:02:14.225 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38814.log 2024-03-29T11:02:14.225 INFO:teuthology.orchestra.run.smithi112.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39243.log.gz 2024-03-29T11:02:14.225 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39673.log 2024-03-29T11:02:14.225 INFO:teuthology.orchestra.run.smithi112.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38463.log.gz 2024-03-29T11:02:14.225 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38970.log 2024-03-29T11:02:14.225 INFO:teuthology.orchestra.run.smithi112.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38580.log.gz 2024-03-29T11:02:14.225 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39673.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38814.log.gz 2024-03-29T11:02:14.225 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40224.log 2024-03-29T11:02:14.225 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39633.log 2024-03-29T11:02:14.225 INFO:teuthology.orchestra.run.smithi112.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39673.log.gz 2024-03-29T11:02:14.225 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.40224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39360.log 2024-03-29T11:02:14.225 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38970.log.gz 2024-03-29T11:02:14.225 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38931.log 2024-03-29T11:02:14.226 INFO:teuthology.orchestra.run.smithi112.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40224.log.gz 2024-03-29T11:02:14.226 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39399.log 2024-03-29T11:02:14.226 INFO:teuthology.orchestra.run.smithi112.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39633.log.gz 2024-03-29T11:02:14.226 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39869.log 2024-03-29T11:02:14.226 INFO:teuthology.orchestra.run.smithi112.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39360.log.gz 2024-03-29T11:02:14.226 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39399.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38931.log.gz 2024-03-29T11:02:14.226 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39555.log 2024-03-29T11:02:14.226 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39908.log 2024-03-29T11:02:14.226 INFO:teuthology.orchestra.run.smithi112.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39399.log.gz 2024-03-29T11:02:14.226 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40103.log 2024-03-29T11:02:14.226 INFO:teuthology.orchestra.run.smithi112.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39869.log.gz 2024-03-29T11:02:14.226 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39908.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39555.log.gz 2024-03-29T11:02:14.226 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40263.log 2024-03-29T11:02:14.226 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.40103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40692.log 2024-03-29T11:02:14.227 INFO:teuthology.orchestra.run.smithi112.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39908.log.gz 2024-03-29T11:02:14.227 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.40263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40887.log 2024-03-29T11:02:14.227 INFO:teuthology.orchestra.run.smithi112.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40103.log.gz 2024-03-29T11:02:14.227 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.40692.log: gzip 85.8% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.40263.log.gz 2024-03-29T11:02:14.227 INFO:teuthology.orchestra.run.smithi112.stderr: -- /var/log/ceph/ceph-client.admin.34797.log 2024-03-29T11:02:14.227 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.40887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38305.log 2024-03-29T11:02:14.227 INFO:teuthology.orchestra.run.smithi112.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40692.log.gz 2024-03-29T11:02:14.227 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.34797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40536.log 2024-03-29T11:02:14.227 INFO:teuthology.orchestra.run.smithi112.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40887.log.gz 2024-03-29T11:02:14.227 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40731.log 2024-03-29T11:02:14.227 INFO:teuthology.orchestra.run.smithi112.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34797.log.gz 2024-03-29T11:02:14.227 INFO:teuthology.orchestra.run.smithi112.stderr: 85.7%/var/log/ceph/ceph-client.admin.40536.log: -- replaced with /var/log/ceph/ceph-client.admin.38305.log.gz 2024-03-29T11:02:14.227 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39048.log 2024-03-29T11:02:14.227 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.40731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40926.log 2024-03-29T11:02:14.228 INFO:teuthology.orchestra.run.smithi112.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40536.log.gz 2024-03-29T11:02:14.228 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38697.log 2024-03-29T11:02:14.228 INFO:teuthology.orchestra.run.smithi112.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40731.log.gz 2024-03-29T11:02:14.228 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.40926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38188.log 2024-03-29T11:02:14.228 INFO:teuthology.orchestra.run.smithi112.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39048.log.gz 2024-03-29T11:02:14.228 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38697.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40926.log.gz 2024-03-29T11:02:14.228 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41136.log 2024-03-29T11:02:14.228 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38149.log 2024-03-29T11:02:14.228 INFO:teuthology.orchestra.run.smithi112.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38697.log.gz 2024-03-29T11:02:14.228 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.41136.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38188.log.gz 2024-03-29T11:02:14.228 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39830.log 2024-03-29T11:02:14.228 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39438.log 2024-03-29T11:02:14.228 INFO:teuthology.orchestra.run.smithi112.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41136.log.gz 2024-03-29T11:02:14.228 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41057.log 2024-03-29T11:02:14.228 INFO:teuthology.orchestra.run.smithi112.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38149.log.gz 2024-03-29T11:02:14.229 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39438.log: gzip -5 87.1% --verbose -- /var/log/ceph/ceph-client.admin.41097.log -- replaced with /var/log/ceph/ceph-client.admin.39830.log.gz 2024-03-29T11:02:14.229 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-03-29T11:02:14.229 INFO:teuthology.orchestra.run.smithi112.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39438.log.gz 2024-03-29T11:02:14.229 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:02:14.229 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.41057.log: /var/log/ceph/ceph-client.admin.41097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41580.log 2024-03-29T11:02:14.229 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40380.log 2024-03-29T11:02:14.229 INFO:teuthology.orchestra.run.smithi112.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41057.log.gz 2024-03-29T11:02:14.229 INFO:teuthology.orchestra.run.smithi112.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41097.log.gz 2024-03-29T11:02:14.229 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.41580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40770.log 2024-03-29T11:02:14.229 INFO:teuthology.orchestra.run.smithi112.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:02:14.229 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.40380.log: gzip -5 86.1% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.41580.log.gz /var/log/ceph/ceph-client.admin.34512.log 2024-03-29T11:02:14.229 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-03-29T11:02:14.229 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40809.log 2024-03-29T11:02:14.229 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.40770.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40380.log.gz 2024-03-29T11:02:14.229 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.34512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38775.log 2024-03-29T11:02:14.230 INFO:teuthology.orchestra.run.smithi112.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40770.log.gz 2024-03-29T11:02:14.230 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.40809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38736.log 2024-03-29T11:02:14.230 INFO:teuthology.orchestra.run.smithi112.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34512.log.gz 2024-03-29T11:02:14.230 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40497.log 2024-03-29T11:02:14.230 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38775.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40809.log.gz 2024-03-29T11:02:14.230 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39986.log 2024-03-29T11:02:14.230 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.40497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41175.log 2024-03-29T11:02:14.230 INFO:teuthology.orchestra.run.smithi112.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38775.log.gz 2024-03-29T11:02:14.230 INFO:teuthology.orchestra.run.smithi112.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38736.log.gz 2024-03-29T11:02:14.230 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39986.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:02:14.230 INFO:teuthology.orchestra.run.smithi112.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40497.log.gz 2024-03-29T11:02:14.230 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.41175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38619.log 2024-03-29T11:02:14.230 INFO:teuthology.orchestra.run.smithi112.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39986.log.gz 2024-03-29T11:02:14.230 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38502.log 2024-03-29T11:02:14.231 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-osd.1.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41175.log.gz 2024-03-29T11:02:14.231 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40025.log 2024-03-29T11:02:14.231 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40653.log 2024-03-29T11:02:14.231 INFO:teuthology.orchestra.run.smithi112.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38619.log.gz 2024-03-29T11:02:14.231 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.40025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41018.log 2024-03-29T11:02:14.231 INFO:teuthology.orchestra.run.smithi112.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38502.log.gz 2024-03-29T11:02:14.231 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.40653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40302.log 2024-03-29T11:02:14.231 INFO:teuthology.orchestra.run.smithi112.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40025.log.gz 2024-03-29T11:02:14.231 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.41018.log: gzip -5 86.7% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.40653.log.gz 2024-03-29T11:02:14.231 INFO:teuthology.orchestra.run.smithi112.stderr: /var/log/ceph/ceph-client.admin.38385.log 2024-03-29T11:02:14.231 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.40302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39087.log 2024-03-29T11:02:14.231 INFO:teuthology.orchestra.run.smithi112.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41018.log.gz 2024-03-29T11:02:14.231 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38385.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40302.log.gz 2024-03-29T11:02:14.231 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39204.log 2024-03-29T11:02:14.232 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40142.log 2024-03-29T11:02:14.232 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39087.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38385.log.gz 2024-03-29T11:02:14.232 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38541.log 2024-03-29T11:02:14.232 INFO:teuthology.orchestra.run.smithi112.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39087.log.gz 2024-03-29T11:02:14.232 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.40142.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39204.log.gz 2024-03-29T11:02:14.232 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41385.log 2024-03-29T11:02:14.232 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40185.log 2024-03-29T11:02:14.232 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38541.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40142.log.gz 2024-03-29T11:02:14.232 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.41385.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:02:14.232 INFO:teuthology.orchestra.run.smithi112.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.38541.log.gz 2024-03-29T11:02:14.232 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40419.log 2024-03-29T11:02:14.232 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.40185.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41385.log.gz 2024-03-29T11:02:14.232 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39751.log 2024-03-29T11:02:14.232 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.40419.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40185.log.gz 2024-03-29T11:02:14.232 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38424.log 2024-03-29T11:02:14.233 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39751.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40419.log.gz 2024-03-29T11:02:14.233 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39712.log 2024-03-29T11:02:14.233 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39516.log 2024-03-29T11:02:14.233 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38424.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39751.log.gz 2024-03-29T11:02:14.233 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41463.log 2024-03-29T11:02:14.233 INFO:teuthology.orchestra.run.smithi112.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38424.log.gz 2024-03-29T11:02:14.233 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39516.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:02:14.233 INFO:teuthology.orchestra.run.smithi112.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39712.log.gz 2024-03-29T11:02:14.233 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.41463.log: gzip -5 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39516.log.gz 2024-03-29T11:02:14.233 INFO:teuthology.orchestra.run.smithi112.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38344.log 2024-03-29T11:02:14.233 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38266.log 2024-03-29T11:02:14.233 INFO:teuthology.orchestra.run.smithi112.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41463.log.gz 2024-03-29T11:02:14.233 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39477.log 2024-03-29T11:02:14.233 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41541.log 2024-03-29T11:02:14.234 INFO:teuthology.orchestra.run.smithi112.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38344.log.gz 2024-03-29T11:02:14.234 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39477.log: gzip 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38266.log.gz 2024-03-29T11:02:14.234 INFO:teuthology.orchestra.run.smithi112.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40965.log 2024-03-29T11:02:14.234 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.41541.log: gzip -5 --verbose 87.2% -- /var/log/ceph/ceph-client.admin.39126.log 2024-03-29T11:02:14.234 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39477.log.gz 2024-03-29T11:02:14.234 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.40965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39321.log 2024-03-29T11:02:14.234 INFO:teuthology.orchestra.run.smithi112.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41541.log.gz 2024-03-29T11:02:14.234 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose 85.8%/var/log/ceph/ceph-client.admin.39126.log: -- replaced with /var/log/ceph/ceph-client.admin.40965.log.gz 2024-03-29T11:02:14.234 INFO:teuthology.orchestra.run.smithi112.stderr: -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:02:14.234 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41424.log 2024-03-29T11:02:14.234 INFO:teuthology.orchestra.run.smithi112.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39126.log.gz 2024-03-29T11:02:14.234 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-osd.0.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39321.log.gz 2024-03-29T11:02:14.234 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34900.log 2024-03-29T11:02:14.235 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.41424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39009.log 2024-03-29T11:02:14.235 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.34900.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41424.log.gz 2024-03-29T11:02:14.235 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40064.log 2024-03-29T11:02:14.235 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39594.log 2024-03-29T11:02:14.235 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39009.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34900.log.gz 2024-03-29T11:02:14.235 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.40064.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39009.log.gz 2024-03-29T11:02:14.235 INFO:teuthology.orchestra.run.smithi112.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40064.log.gz 2024-03-29T11:02:14.235 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40458.log 2024-03-29T11:02:14.235 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40341.log 2024-03-29T11:02:14.235 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.40458.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39594.log.gzgzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:02:14.235 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-03-29T11:02:14.235 INFO:teuthology.orchestra.run.smithi112.stderr:gzip 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40458.log.gz 2024-03-29T11:02:14.235 INFO:teuthology.orchestra.run.smithi112.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41307.log 2024-03-29T11:02:14.236 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.40341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38110.log 2024-03-29T11:02:14.236 INFO:teuthology.orchestra.run.smithi112.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40341.log.gz 2024-03-29T11:02:14.236 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.41307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41268.log 2024-03-29T11:02:14.236 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38110.log: 86.2% 92.8% -- replaced with /var/log/ceph/ceph-client.admin.41307.log.gz 2024-03-29T11:02:14.236 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:02:14.236 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:02:14.236 INFO:teuthology.orchestra.run.smithi112.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38110.log.gz 2024-03-29T11:02:14.236 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38892.log 2024-03-29T11:02:14.236 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.41268.log: /var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40575.log 2024-03-29T11:02:14.236 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38892.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41268.log.gz 2024-03-29T11:02:14.236 INFO:teuthology.orchestra.run.smithi112.stderr:gzip 86.0% -5 --verbose -- /var/log/ceph/ceph-client.admin.39165.log 2024-03-29T11:02:14.236 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38892.log.gz 2024-03-29T11:02:14.236 INFO:teuthology.orchestra.run.smithi112.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:02:14.236 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.40575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40848.log 2024-03-29T11:02:14.237 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39165.log: 86.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39282.log 2024-03-29T11:02:14.237 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40575.log.gz 2024-03-29T11:02:14.237 INFO:teuthology.orchestra.run.smithi112.stderr: 87.1%/var/log/ceph/ceph-client.admin.40848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41502.log 2024-03-29T11:02:14.237 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39165.log.gz 2024-03-29T11:02:14.237 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39282.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40848.log.gz 2024-03-29T11:02:14.237 INFO:teuthology.orchestra.run.smithi112.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39282.log.gz 2024-03-29T11:02:14.237 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39791.log 2024-03-29T11:02:14.237 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.41502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38227.log 2024-03-29T11:02:14.237 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.39791.log: 86.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41346.log 2024-03-29T11:02:14.237 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41502.log.gz 2024-03-29T11:02:14.237 INFO:teuthology.orchestra.run.smithi112.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39791.log.gz 2024-03-29T11:02:14.237 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38227.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38227.log.gz 2024-03-29T11:02:14.237 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40614.log 2024-03-29T11:02:14.237 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.41346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38658.log 2024-03-29T11:02:14.237 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.40614.log: 86.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41229.log 2024-03-29T11:02:14.238 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41346.log.gz 2024-03-29T11:02:14.238 INFO:teuthology.orchestra.run.smithi112.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40614.log.gz 2024-03-29T11:02:14.238 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.38658.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:02:14.238 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.41229.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38658.log.gz 2024-03-29T11:02:14.238 INFO:teuthology.orchestra.run.smithi112.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41229.log.gz 2024-03-29T11:02:14.276 INFO:teuthology.orchestra.run.smithi118.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:02:25.983 INFO:teuthology.orchestra.run.smithi118.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:02:26.071 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-mon.c.log: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:02:26.105 INFO:teuthology.orchestra.run.smithi112.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:02:26.605 INFO:teuthology.orchestra.run.smithi118.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:02:30.104 INFO:teuthology.orchestra.run.smithi112.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:02:30.214 INFO:teuthology.orchestra.run.smithi112.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:02:30.556 INFO:teuthology.orchestra.run.smithi118.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:02:30.669 INFO:teuthology.orchestra.run.smithi118.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:03:04.123 INFO:teuthology.orchestra.run.smithi112.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:03:09.269 INFO:teuthology.orchestra.run.smithi112.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:03:09.271 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-03-29T11:03:09.271 INFO:teuthology.orchestra.run.smithi112.stderr:real 0m55.118s 2024-03-29T11:03:09.271 INFO:teuthology.orchestra.run.smithi112.stderr:user 2m34.965s 2024-03-29T11:03:09.272 INFO:teuthology.orchestra.run.smithi112.stderr:sys 0m5.489s 2024-03-29T11:03:19.859 INFO:teuthology.orchestra.run.smithi118.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:03:19.860 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-29T11:03:19.861 INFO:teuthology.orchestra.run.smithi118.stderr:real 1m5.671s 2024-03-29T11:03:19.861 INFO:teuthology.orchestra.run.smithi118.stderr:user 1m58.043s 2024-03-29T11:03:19.861 INFO:teuthology.orchestra.run.smithi118.stderr:sys 0m4.457s 2024-03-29T11:03:19.861 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:03:19.862 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622657/remote/smithi112/log 2024-03-29T11:03:19.863 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:03:42.253 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622657/remote/smithi118/log 2024-03-29T11:03:42.254 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:03:59.883 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:04:00.090 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e6f403dca5384055a0618a7703588c0e Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:04:00.094 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:04:00.107 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:04:00.115 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:04:00.115 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:04:00.129 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:04:00.211 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:04:00.219 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:04:00.219 DEBUG:teuthology.orchestra.run.smithi112:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:04:00.222 DEBUG:teuthology.orchestra.run.smithi118:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:04:00.236 INFO:teuthology.orchestra.run.smithi112.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:04:00.238 INFO:teuthology.orchestra.run.smithi118.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:04:00.241 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:04:00.241 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2024-03-29T11:04:00.241 INFO:teuthology.orchestra.run.smithi112.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 51 +428us[ +426us] +/- 81ms 2024-03-29T11:04:00.241 INFO:teuthology.orchestra.run.smithi112.stdout:^* hv02.front.sepia.ceph.com 3 6 377 50 -178us[ -180us] +/- 61ms 2024-03-29T11:04:00.241 INFO:teuthology.orchestra.run.smithi112.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 50 -186us[ -188us] +/- 91ms 2024-03-29T11:04:00.241 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:04:00.242 INFO:teuthology.orchestra.run.smithi118.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:04:00.242 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================== 2024-03-29T11:04:00.243 INFO:teuthology.orchestra.run.smithi118.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 50 +445us[ +442us] +/- 81ms 2024-03-29T11:04:00.243 INFO:teuthology.orchestra.run.smithi118.stdout:^* hv02.front.sepia.ceph.com 3 6 377 49 -181us[ -184us] +/- 61ms 2024-03-29T11:04:00.243 INFO:teuthology.orchestra.run.smithi118.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 50 -203us[ -206us] +/- 91ms 2024-03-29T11:04:00.243 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:04:00.244 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:04:00.252 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:04:00.252 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:04:00.278 DEBUG:teuthology.orchestra.run.smithi112:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:04:00.350 DEBUG:teuthology.orchestra.run.smithi118:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:04:00.420 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:04:00.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:04:00.451 DEBUG:teuthology.orchestra.run.smithi118:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:04:00.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:04:00.482 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:04:00.490 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:04:00.498 INFO:teuthology.task.internal:Duration was 1220.734699 seconds 2024-03-29T11:04:00.498 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:04:00.506 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:04:00.507 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:04:00.509 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:04:00.550 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:04:00.564 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:04:00.960 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:04:00.961 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi112.front.sepia.ceph.com 2024-03-29T11:04:00.961 DEBUG:teuthology.orchestra.run.smithi112:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:04:00.981 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi118.front.sepia.ceph.com 2024-03-29T11:04:00.981 DEBUG:teuthology.orchestra.run.smithi118:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:04:01.026 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:04:01.027 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:04:01.029 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:04:01.095 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:04:01.095 DEBUG:teuthology.orchestra.run.smithi112:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:04:01.098 DEBUG:teuthology.orchestra.run.smithi118:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:04:01.232 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:04:01.243 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:04:01.243 DEBUG:teuthology.orchestra.run.smithi112:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:04:01.270 DEBUG:teuthology.orchestra.run.smithi118:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:04:01.297 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:04:01.307 DEBUG:teuthology.orchestra.run.smithi112:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:04:01.312 DEBUG:teuthology.orchestra.run.smithi118:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:04:01.335 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = core 2024-03-29T11:04:01.362 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = core 2024-03-29T11:04:01.386 DEBUG:teuthology.orchestra.run.smithi112:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:04:01.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:04:01.409 DEBUG:teuthology.orchestra.run.smithi118:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:04:01.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:04:01.440 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:04:01.450 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:04:01.451 DEBUG:teuthology.misc:Transferring archived files from smithi112:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622657/remote/smithi112 2024-03-29T11:04:01.452 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:04:01.487 DEBUG:teuthology.misc:Transferring archived files from smithi118:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622657/remote/smithi118 2024-03-29T11:04:01.487 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:04:01.522 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:04:01.522 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:04:01.524 DEBUG:teuthology.orchestra.run.smithi118:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:04:01.571 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:04:01.582 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:04:01.583 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:04:01.596 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:04:01.596 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:04:01.599 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:04:01.610 INFO:teuthology.orchestra.run.smithi112.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:04 /home/ubuntu/cephtest 2024-03-29T11:04:01.626 INFO:teuthology.orchestra.run.smithi118.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:04 /home/ubuntu/cephtest 2024-03-29T11:04:01.627 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:04:01.638 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:04:01.664 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:04:01.690 DEBUG:teuthology.nuke:shortname: smithi112 2024-03-29T11:04:01.690 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:04:01.714 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622657', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:34:18.768824', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHuBDjejHQ6clU+zE35ohc++KSsObk4DDq5tqpmPWL13rsLQqLqicbLxqFFy7qS/SeaQoaT9UKuFHowzwksXC5w='} 2024-03-29T11:04:01.717 DEBUG:teuthology.nuke:shortname: smithi118 2024-03-29T11:04:01.718 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:04:01.741 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622657', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:34:18.766910', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJLpc8+kcufOIZ99A3ZlOO/UURp3bM65vBveLNBHcmJbrtsmZRV2it8LQ7naivtFd3Kr+YdOsKlHVhkdpc5FJHU='} 2024-03-29T11:04:01.770 INFO:teuthology.orchestra.console.smithi112:Power off 2024-03-29T11:04:01.771 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:04:01.796 DEBUG:teuthology.orchestra.console.smithi112:power off output: Chassis Power Control: Down/Off 2024-03-29T11:04:01.796 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:04:01.813 INFO:teuthology.orchestra.console.smithi118:Power off 2024-03-29T11:04:01.813 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:04:01.824 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is on 2024-03-29T11:04:01.836 DEBUG:teuthology.orchestra.console.smithi118:power off output: Chassis Power Control: Down/Off 2024-03-29T11:04:01.837 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:04:01.860 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is on 2024-03-29T11:04:05.825 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:04:05.862 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:04:05.936 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is on 2024-03-29T11:04:05.972 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is on 2024-03-29T11:04:09.938 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:04:09.974 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:04:10.049 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is on 2024-03-29T11:04:10.084 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is off 2024-03-29T11:04:10.185 INFO:teuthology.orchestra.console.smithi118:Power off completed 2024-03-29T11:04:14.054 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:04:14.168 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is off 2024-03-29T11:04:14.269 INFO:teuthology.orchestra.console.smithi112:Power off completed 2024-03-29T11:04:14.374 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1220.7346994876862 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e6f403dca5384055a0618a7703588c0e status: fail success: false 2024-03-29T11:04:14.374 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:04:14.466 INFO:teuthology.run:FAIL